Key takeaways

  • Attention scores every query against every key, turns the scores into weights with a softmax and averages the values: softmax(QKᵀ / √d_k)V.
  • Bahdanau, Cho and Bengio introduced it in 2014 to remove the fixed-length vector bottleneck in translation, and the 2017 transformer built a whole model from it.
  • Its cost grows with the square of the input length, and later work such as grouped-query attention and FlashAttention cuts memory traffic without changing the core idea.

The attention mechanism is the part of a neural network that lets each word look at every other word in the input and decide how much each one matters. It turns those judgments into weights and takes a weighted average of the other words' information. The transformer, the design behind most modern language models, is built around it.

It started in 2014 as a fix for machine translation, became a whole architecture in 2017, and has since been reworked to make it cheaper to run.

How the attention mechanism works

Take "the animal didn't cross the street because it was tired." To understand "it", a model has to work out that it means the animal. Attention does that.

The 2017 paper Attention Is All You Need defines attention as mapping "a query and a set of key-value pairs to an output." Each word becomes three learned vectors, lists of numbers:

  • A query: what this word is looking for.
  • A key: what this word offers to match against.
  • A value: the information it passes on if picked.

The query of "it" is compared with the key of every word. Good matches score high, the scores become weights that sum to 1, and the output is the values blended by those weights. If "animal" scored highest, the new "it" carries mostly the animal's information. The model learns during training how to make queries, keys and values.

The attention mechanism formula

The paper writes it in one line, called scaled dot-product attention:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) V
  1. QKᵀ compares every query with every key. For n words that gives an n by n table of scores.
  2. / √d_k divides by the square root of the key size. The paper says large dot products push the softmax into regions with extremely small gradients, which stalls learning.
  3. softmax turns each row of scores into weights between 0 and 1.
  4. × V averages the values with those weights.

The older alternative, additive attention, scores each pair with a small neural network. The authors chose dot products because "dot-product attention is much faster and more space-efficient in practice."

Where attention came from

Translation models in 2014 used recurrent neural networks (RNNs), which read one word at a time and squeezed the whole sentence into one fixed-length vector. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio argued in Neural Machine Translation by Jointly Learning to Align and Translate, posted September 1, 2014 and presented at ICLR 2015, that this vector was the bottleneck.

Their fix: for each word it writes, the decoder scores every source word, turns the scores into weights with a softmax and builds a fresh weighted summary. Older models got worse as sentences got longer. Theirs, the paper says, "shows no performance deterioration even with sentences of length 50 or more."

The transformer kept attention and dropped the RNN entirely. Our transformer model explainer covers the full architecture.

Self-attention vs cross-attention

The difference is where queries, keys and values come from. The transformer uses attention three ways:

Type Queries from Keys and values from
Self-attention The input The same input
Masked self-attention The text written so far The same text, earlier tokens only
Cross-attention The decoder The encoder's output

Self-attention works within one sequence. Cross-attention links two, as when a translation decoder consults the encoded source sentence. The mask stops a token from seeing tokens after it, which lets a model generate left to right.

Attention also ignores word order on its own: "dog bites man" and "man bites dog" look alike. Models add position signals, covered in our positional encoding explainer.

What multi-head attention adds

A word relates to others in several ways at once: "it" needs its noun, a verb needs its subject. Multi-head attention runs several attention operations side by side, each with its own learned projections, then joins the results. The paper says this lets the model attend to "different representation subspaces at different positions." The base model used 8 heads of 64 dimensions each, at about the cost of one full-size head.

Why attention is quadratic

Every query meets every key, so n tokens need n × n comparisons. The paper gives a self-attention layer's cost as O(n²·d), where d is the vector size. Double the text and that work roughly quadruples. The trade was worth it in 2017: unlike an RNN, attention connects all positions in a constant number of steps, so it trains in parallel.

At long inputs the n² term dominates, which is a big reason long context windows are expensive.

The KV cache and slow generation

A model generates one token at a time, and each new token attends to all earlier ones. Their keys and values don't change, so implementations store them instead of recomputing them. That store is the KV cache. Noam Shazeer's 2019 paper Fast Transformer Decoding: One Write-Head is All You Need shows it in code: each step takes the previous keys and values and appends the new ones.

The catch is that the whole cache is read from memory at every step. Shazeer found decoding speed is limited by the memory bandwidth needed to reload those tensors. In his TPUv2 tests, the baseline decoder took 46 microseconds per token, against 1.7 for the encoder, which handles all positions at once. Speculative decoding attacks the same cost from another angle.

Multi-query and grouped-query attention

Standard multi-head attention gives every head its own keys and values, so the cache grows with the head count. Multi-query attention, from Shazeer's paper, shares one set across all heads. Decoding fell from 46 to 3.8 microseconds per token, with what he calls minor quality loss.

Grouped-query attention (GQA), from Ainslie et al. (EMNLP 2023), is the middle ground: query heads are split into groups, and each group shares one key and value head. The authors converted existing T5 models by averaging key and value heads, then trained for 5% of the original pre-training steps. On T5-XXL, on one TPUv4 chip:

Model Time per sample Average score
Multi-head 1.51 s 47.2
Multi-query 0.24 s 46.6
Grouped-query, 8 groups 0.28 s 47.1

They tested only encoder-decoder models, not the decoder-only kind used for chat.

FlashAttention: same math, less memory traffic

FlashAttention, by Tri Dao and colleagues in May 2022, changes how attention runs on a GPU, not what it computes. The output is exact.

A GPU has large, slow main memory and small, fast on-chip memory. FlashAttention processes queries, keys and values in tiles inside the fast memory and never writes the full n by n score table to the slow one. Its extra memory grows linearly with input length.

It does more arithmetic but runs faster. On GPT-2 medium the paper reports 75.2 GFLOPs against 66.6 for standard attention, but memory traffic fell from 40.3 GB to 4.4 GB and runtime from 41.7 ms to 7.3 ms. It doesn't remove the n² comparisons. It makes them cheaper.

Why it matters

The core has barely changed since 2017: queries, keys, values, a softmax and a weighted sum. The work since then has been on cost. The KV cache trades memory for speed, grouped-query attention shrinks that cache, and FlashAttention cuts memory traffic. That is what makes long inputs affordable to serve.

Frequently asked questions

What is the difference between self-attention and cross-attention?

In self-attention, the queries, keys and values all come from the same sequence, so words in a text attend to each other. In cross-attention, the queries come from one sequence and the keys and values from another, as when a translation decoder attends to the encoded source sentence.

What is multi-head attention?

It runs several attention operations in parallel, each with its own learned projections of the queries, keys and values, then joins the results. The original transformer used 8 heads of 64 dimensions each, so different heads can track different relationships at the same cost as one full-size head.

Why is attention quadratic?

Every token's query is compared with every token's key, so n tokens need n × n comparisons. The transformer paper gives the cost of a self-attention layer as O(n²·d), which means doubling the input roughly quadruples that work.

What is the attention mechanism formula?

Scaled dot-product attention is Attention(Q, K, V) = softmax(QKᵀ / √d_k)V. The query-key dot products are divided by the square root of the key size so the softmax doesn't saturate, then the softmax weights average the values.

Sources

  1. Attention Is All You NeedarXiv
  2. Neural Machine Translation by Jointly Learning to Align and TranslatearXiv
  3. Fast Transformer Decoding: One Write-Head is All You NeedarXiv
  4. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsarXiv
  5. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessarXiv

Add ClaudeGPT Si as a preferred source on Google