Natural Language Processing & Transformers

From Word Embeddings to Multi-Head Self-Attention

« Computer Vision & Convolutional Neural Networks (CNNs)« Previous Reinforcement Learning & Policy Optimization »Next »

Natural Language Processing (NLP) enables machines to read, synthesize, and generate human languages. The field was revolutionized by the 2017 Transformer architecture.

Evolution from Recurrence to Self-Attention

Early NLP relied on Recurrent Neural Networks (RNNs) and LSTMs that processed sentences sequentially token-by-token. This caused catastrophic forgetting over long paragraphs and prevented GPU parallelization.

The Transformer (introduced in "Attention Is All You Need") replaced recurrence with Self-Attention, allowing every word in a sequence to attend to every other word simultaneously.

Tokens are projected into Queries (Q), Keys (K), and Values (V). Scaled Dot-Product Attention is computed as: Attention(Q, K, V) = softmax((Q · K^T) / sqrt(d_k)) · V.

Self-Attention Softmax Scoring Concept

Try It Yourself »

import numpy as np

def scaled_dot_product_attention(Q, K, V):
    d_k = Q.shape[-1]
    # Compute attention scores
    scores = np.dot(Q, K.T) / np.sqrt(d_k)
    # Apply softmax normalization across columns
    exp_scores = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
    attention_weights = exp_scores / np.sum(exp_scores, axis=-1, keepdims=True)
    # Output weighted sum of values
    return np.dot(attention_weights, V), attention_weights

# 2 tokens with 3 embedding dimensions
Q = np.array([[1.0, 0.5, 0.0], [0.0, 0.8, 1.2]])
K = Q.copy()
V = np.array([[10.0, 20.0], [30.0, 40.0]])

context, weights = scaled_dot_product_attention(Q, K, V)
print("Contextual representation shape:", context.shape)
print("Attention weights matrix:
", weights)

Self-attention determines dynamic semantic weight between pairs of words in a sentence.

Google AdSense Slot (inContentBanner)

Set your Publisher ID in src/config/ads.ts to display live ads.

Test Yourself With Exercise

+15 XP

What fundamental breakthrough did the Transformer architecture introduce over traditional RNNs?

« Previous: Computer Vision & Convolutional Neural Networks (CNNs)Next: Reinforcement Learning & Policy Optimization »