Transformer Architecture: A Visual & Interactive Guide

The Problem

Computers don't understand words — they understand numbers. But how do we convert "cat" into something a neural network can process?

Naive approach: Assign each word a number (cat=1, dog=2, …)

• Problem: This implies "cat" and "dog" are as different as "cat" and "quantum physics"
• We lose all semantic meaning

Better approach: Represent each word as a vector in high-dimensional space where:

• Similar words are close together
• Relationships are encoded as directions

Key Insight: The King–Queen Analogy

The famous example from the video:

This works because embeddings capture relationships:

• The vector from "man" to "woman" represents a gender direction
• Apply that same direction to "king" and you get "queen"

The Math

For a vocabulary of V words and embedding dimension d:

To get the embedding for token i:

It's just a lookup table! Each row is a learnable d-dimensional vector.

Code: Exploring Real Embeddings

# Using sentence-transformers for easy embedding access
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# Get embeddings for words
words = ["king", "queen", "man", "woman", "prince", "princess"]
embeddings = model.encode(words)

# Check dimensions
print(f"Embedding shape: {embeddings[0].shape}")  # 384 dimensions

# Compute cosine similarity
def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# King is more similar to queen than to random words
print(f"king-queen similarity: {cosine_sim(embeddings[0], embeddings[1]):.3f}")
print(f"king-man similarity: {cosine_sim(embeddings[0], embeddings[2]):.3f}")

# Try the analogy: king - man + woman ≈ queen
analogy = embeddings[0] - embeddings[2] + embeddings[3]  # king - man + woman
print(f"Analogy result similarity to queen: {cosine_sim(analogy, embeddings[1]):.3f}")

Key Takeaways

The Problem

Consider these sentences:

• "The cat ate the fish"
• "The fish ate the cat"

Same words, completely different meanings! But if we just use embeddings, the transformer sees the same set of vectors (in different positions). Unlike RNNs that process sequentially, transformers process all tokens in parallel — they have no inherent notion of order.

The Solution: Add Position Information

We need to inject position into each token's representation. The original transformer uses sinusoidal positional encodings:

Where:

• pos = position in sequence (0, 1, 2, …)
• i = dimension index
• d = embedding dimension

Why not just use position numbers (0, 1, 2, …)?

• Numbers would dominate the embedding values
• No natural way to handle sequences longer than training data
• Sinusoids can extrapolate to unseen positions

Why alternating sin/cos?

• Allows the model to learn relative positions
• PE_pos+k can be represented as a linear function of PE_pos

Code: Visualizing Positional Encodings

import numpy as np
import matplotlib.pyplot as plt

def positional_encoding(max_len, d_model):
    """Generate sinusoidal positional encodings."""
    pe = np.zeros((max_len, d_model))
    position = np.arange(max_len)[:, np.newaxis]

    # Different frequencies for different dimensions
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))

    pe[:, 0::2] = np.sin(position * div_term)  # Even dimensions
    pe[:, 1::2] = np.cos(position * div_term)  # Odd dimensions

    return pe

# Generate encodings
pe = positional_encoding(max_len=100, d_model=64)

# Visualize as heatmap
plt.figure(figsize=(12, 4))
plt.imshow(pe.T, aspect='auto', cmap='RdBu')
plt.xlabel('Position')
plt.ylabel('Dimension')
plt.title('Positional Encodings: Each column is a unique fingerprint')
plt.colorbar()
plt.show()

# Show how position 0 vs position 50 differ
print("Position 0 (first 10 dims):", pe[0, :10].round(3))
print("Position 50 (first 10 dims):", pe[50, :10].round(3))

Modern Alternatives

Key Takeaways

The Problem: Context Matters

Consider the word "bank":

• "I deposited money at the bank" → financial institution
• "I sat on the river bank" → edge of water

The same word needs different representations depending on context. Self-attention solves this by letting each token look at all other tokens to build a context-aware representation.

The Core Idea: Questions, Keys, and Values

Think of attention like a search engine:

Each token generates all three:

• Its Query: What it's looking for
• Its Key: What it offers to others
• Its Value: The information it provides

The Math

Given input X of shape (sequence_length, d_model):

Step 1: Create Q, K, V

Step 2: Compute attention scores

Step 3: Scale and normalize

Step 4: Weighted sum of values

Code: Implementing Self-Attention from Scratch

import numpy as np

def softmax(x, axis=-1):
    """Numerically stable softmax."""
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

def self_attention(X, W_q, W_k, W_v):
    """
    Single-head self-attention.

    X: (seq_len, d_model) - input embeddings
    W_q, W_k, W_v: (d_model, d_k) - projection matrices
    """
    # Step 1: Project to Q, K, V
    Q = X @ W_q  # (seq_len, d_k)
    K = X @ W_k  # (seq_len, d_k)
    V = X @ W_v  # (seq_len, d_k)

    # Step 2: Compute attention scores
    d_k = K.shape[-1]
    scores = Q @ K.T  # (seq_len, seq_len)

    # Step 3: Scale and softmax
    scaled_scores = scores / np.sqrt(d_k)
    attention_weights = softmax(scaled_scores, axis=-1)

    # Step 4: Weighted sum of values
    output = attention_weights @ V  # (seq_len, d_k)

    return output, attention_weights

# Example: 4 tokens, 8-dimensional embeddings, 4-dimensional attention
np.random.seed(42)
seq_len, d_model, d_k = 4, 8, 4

X = np.random.randn(seq_len, d_model)
W_q = np.random.randn(d_model, d_k) * 0.1
W_k = np.random.randn(d_model, d_k) * 0.1
W_v = np.random.randn(d_model, d_k) * 0.1

output, attn_weights = self_attention(X, W_q, W_k, W_v)

print("Attention weights (each row sums to 1):")
print(attn_weights.round(3))
print("\nRow sums:", attn_weights.sum(axis=1).round(3))

Key Takeaways

The Problem: Raw Scores Are Messy

After computing QK^T, we get raw attention scores:

• Can be any real number (positive or negative)
• Don't sum to anything meaningful
• Larger scores = more relevance, but how much more?

We need a way to convert these to proper attention weights:

• Non-negative (can't have negative attention)
• Sum to 1 (it's an "attention budget")
• Higher scores → higher weights

Softmax: The Solution

Why this works:

1. Exponential makes everything positive: e^x > 0 for all x
2. Division makes them sum to 1: Proper probability distribution
3. Amplifies differences: Small score differences become large weight differences

The Temperature Parameter

You can control the "sharpness" of softmax with temperature:

This is the same "temperature" parameter you see in ChatGPT!

The Scaling Factor: √d_k

Why do we divide by √d_k before softmax?

The problem: Dot products get larger as dimension increases.

• Q and K are d_k-dimensional vectors
• Their dot product is sum of d_k terms
• Variance grows with d_k

Large dot products → extreme softmax:

• Scores like [50, 52, 48] → softmax ≈ [0.1, 0.8, 0.1]
• Scores like [500, 520, 480] → softmax ≈ [0.0, 1.0, 0.0]

The model gets overconfident and can't learn from gradients!

Solution: Divide by √d_k to normalize variance back to ~1.

Key Takeaways

The Problem: One Head Isn't Enough

A single attention head can only focus on one type of relationship at a time. But language has many simultaneous relationships:

• Syntactic: subject-verb agreement
• Semantic: word meaning in context
• Coreference: what "it" refers to
• Positional: nearby words

The Solution: Multiple Heads in Parallel

Instead of one big attention operation, run several smaller ones:

Where each head is:

How It Works

1. Split the embedding into h heads (e.g., 768 dimensions → 12 heads of 64 each)
2. Compute attention independently in each head
3. Concatenate the results back together
4. Project through a final linear layer

Each head learns to focus on different types of relationships!

What Different Heads Learn

Research has shown that heads specialize:

Key takeaways

The Full Architecture

Input
  │
  ├──────────────────┐
  │                  │
  ▼                  │
Layer Norm           │
  │                  │
  ▼                  │
Multi-Head Attention │
  │                  │
  ▼                  │
  + ◄────────────────┘  (Residual connection)
  │
  ├──────────────────┐
  │                  │
  ▼                  │
Layer Norm           │
  │                  │
  ▼                  │
Feed-Forward Network │
  │                  │
  ▼                  │
  + ◄────────────────┘  (Residual connection)
  │
  ▼
Output

Summary: What Each Component Does

Now that you understand the fundamentals:

1. Implement a mini-transformer from scratch in PyTorch
2. Explore attention visualizations with BertViz or similar tools
3. Fine-tune a pretrained model on a task you care about
4. Read "Attention Is All You Need" — it'll make much more sense now!
5. Explore variations: RoPE, ALiBi, Flash Attention, sparse attention