Transformers — How AI Works

The Hook: It's Just Layers

Everything we've covered so far—tokenization, embeddings, attention—comes together in the transformer. A transformer is a stack of layers. Each layer takes in a sequence of vectors, applies attention, then feedforward processing, and outputs a new sequence of vectors. Stack enough of these, and the model can do remarkable things.

There's no magic here. No understanding in the human sense. Just: learned representations, stacked and composed, trained on massive data to predict the next token. That's all a language model is. The transformer architecture is what makes that training tractable.

Key Insight

A transformer is a feedforward block + attention block, repeated N times. Each layer refines the representations. Depth (more layers) beats width (more neurons per layer). GPT-3 has 96 layers. That's it.

The Architecture

A transformer layer has two sub-blocks:

1. Multi-head self-attention: Tokens attend to other tokens in the same layer. The output is a context-enriched sequence.

2. Feedforward network: A standard two-layer neural network applied position-by-position. Same weights shared across all positions (but different weights than attention).

Each sub-block has a residual connection (skip connection) and layer normalization. These stabilize training and enable gradient flow through very deep networks.

SINGLE TRANSFORMER LAYER Input: sequence of vectors │ ┌──────┴──────┐ │ │ ┌─▼────────────▼─┐ │ MULTI-HEAD │ │ SELF-ATTENTION │ └─┬────────────┬─┘ │Add & LayerNorm│ │ │ ┌─▼────────────▼─┐ │ FEEDFORWARD │ │ NETWORK │ └─┬────────────┬─┘ │Add & LayerNorm│ ▼ ▼ Output: refined sequence Repeat N times (N = layer count)

The Stack: Layer by Layer

Here's what happens across a transformer stack. Research on intermediate layers (BERT-style probing) shows learned representations evolve systematically:

Layer 1-4

Surface features

Token-level patterns, positional info, word-level statistics

Layer 5-12

Syntactic patterns

POS tagging,依存句法, coreference clusters emerge

Layer 13-24

Semantic relationships

Entity relations, world knowledge, semantic similarity

Layer 25-96

Task-specific reasoning

In-context learning, multi-step reasoning, output formatting

Early layers preserve surface form. Middle layers extract structure. Late layers perform computation. This hierarchical composition is what makes transformers generalize: the same building blocks recombine to handle novel inputs.

How Training Works

Transformers are trained with self-supervised learning on text. The task: predict the next token given all previous tokens. This is called causal language modeling (CLM) or autoregressive modeling.

// Autoregressive training (simplified)
for each training_sequence [t1, t2, t3, ..., tn]:
    // Input: all tokens up to position i
    input = [t1, t2, ..., t_(i-1)]
    
    // Forward pass through all N layers
    output = transformer(input)
    
    // Predict next token: softmax over vocabulary
    predicted_prob = softmax(output[-1])
    
    // Loss: cross-entropy against actual next token
    loss = -log(predicted_prob[ti])
    
    // Backprop: update all weights via gradient descent
    loss.backward()
    optimizer.step()
    
// Trillions of tokens × billions of parameters = GPT-4
    

The key insight: you don't need labeled data. Text is labels. The next token is the label. Just keep predicting the next token across billions of examples, and the network learns to build useful internal representations.

Scale: The Numbers

176B

GPT-4 parameters

~$100M

estimated training cost

15T

tokens in training

layers (GPT-3)

Scale is not incidental. It's the primary driver of capability. More parameters + more data + more compute = better performance on nearly every benchmark. This relationship (the "scaling laws") was discovered empirically and has held across model generations. We don't fully understand why it works so well, but it does.

Decoder-Only vs Full Transformer

The original "Attention Is All You Need" paper described an encoder-decoder architecture (for translation). Most modern LLMs are decoder-only (GPT architecture). The difference:

Encoder: Sees the full sequence bidirectionally. Each token attends to all others. Good for understanding tasks (classification, NER).

Decoder: Causal mask. Each token only attends to previous tokens. Can't see future. Good for generation. What GPT uses.

Encoder-Decoder: Both. Used by T5, BART, and some translation models. Less common now.

Positional Encoding: The Missing Piece

Attention is permutation-invariant: "the dog bites" and "the bites dog" produce the same attention patterns without position information. Position is injected via positional encodings: fixed or learned vectors added to token embeddings that encode where in the sequence a token sits.

Modern models (RoPE, ALiBi) use rotary or bias-based approaches that let attention naturally incorporate position without adding to embeddings. This also allows generalization to longer contexts than seen during training (up to a point).

The Key Takeaway

A transformer stacks attention + feedforward layers. Each layer refines token representations through learned attention patterns. Depth builds hierarchy: surface form → syntax → semantics → reasoning. Trained via next-token prediction on massive text. Scale is the primary driver of capability. Everything else follows from that.

← Attention ← All Topics