Every token looks at every other token. That's the trick.
Earlier language models processed tokens in isolation. "Bank" in sentence 5 had no connection to "river" in sentence 1. The model couldn't maintain context across long sequences.
Attention changed everything. Now every token in a sequence can attend to (look at, weigh the importance of) every other token simultaneously. This is the core innovation of the transformer architecture, introduced in "Attention Is All You Need" (2017). It won. Completely.
Attention is a weighted lookup. For each token, the model computes how much it should "pay attention" to every other token, based on learned relevance. The result is a context-aware representation for each position that depends on the full input.
Attention uses three learned projections: Query (Q), Key (K), and Value (V). These are learned during training. For each token, we compute how much it queries other tokens, how much other tokens are "keys" that match that query, and what values should be retrieved.
Step 1: Each token embedding gets projected into Q, K, V vectors via learned weight matrices.
Step 2: For position i, compute scores[i][j] = how much token i queries token j. Dot product of Q_i with K_j.
Step 3: Scale by sqrt(d_k). This prevents softmax saturation (gradients vanishing) for large embedding dimensions.
Step 4: Softmax along each row: converts scores to probabilities that sum to 1. Now "how much attention does i pay to j?" is a probability.
Step 5: Multiply the attention weights by V matrices. Each token output is a weighted blend of all value vectors.
One attention operation is called a head. A transformer runs multiple heads in parallel (typically 12-96). Each head has its own Q, K, V projections. The outputs are concatenated and projected again. This lets the model attend to different aspects of the relationship simultaneously.
Research on analyzing attention heads has shown specific heads specialize: some track subject-verb agreement, some resolve pronouns, some detect synonyms. Different heads learn different linguistic features from the same input.
The softmax at the end of attention computation is critical. Softmax converts arbitrary scores into a probability distribution that sums to 1. This means:
Sparse attention: One token attends strongly to a few relevant tokens, weakly to most others. No binary hard cuts.
Differentiable: Softmax is smooth and differentiable everywhere, so gradients flow through it during backpropagation.
Competitive: Because probabilities sum to 1, paying more attention to one token means paying less to others. This creates meaningful tradeoffs.
Attention computes pairwise scores for every position against every other position. For a sequence of length N, this is O(N²) in memory and compute. For N=10,000 tokens (roughly a short paper), that's 100 million operations. For N=1,000,000 (long document), it's 1 trillion.
This is why context windows are expensive. Extending context requires either more memory, more compute, or approximations. Flash attention, sparse attention, and linear attention variants all attack this problem from different angles.
Attention lets every token dynamically weigh every other token's importance. Q/K/V projections make this learned rather than hardcoded. Multi-head attention runs several attention operations in parallel, capturing diverse relationships. The result: context-aware representations without recurrence, without sequential processing, and with direct access to any token from any position.