Key takeaways

  • A transformer reads all the words of a text in parallel and lets each word weigh every other word through attention.
  • The design came from the 2017 paper 'Attention Is All You Need', which reported 28.4 BLEU on WMT 2014 English-to-German translation.
  • The cost of self-attention grows with the square of sequence length, which is why long context windows are expensive.

A transformer model is a neural network that reads a whole piece of text at once and lets every word look at every other word to work out what it means. Eight Google researchers introduced the design in the 2017 paper Attention Is All You Need, and it is the architecture behind most modern language models.

The idea is easy to state. The model turns text into numbers, then repeatedly asks, for each word, "which other words matter here?" and blends their information in. The rest of this page explains how, using the paper and Google's own write-up of it.

What a transformer model is

The paper, by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser and Illia Polosukhin, first appeared on arXiv on June 12, 2017. Its abstract describes a "sequence transduction model" that relies entirely on attention and drops the two older building blocks, recurrence and convolutions. A sequence transduction model turns one sequence into another, such as an English sentence into a German one.

Before this, the standard tool was the recurrent neural network, or RNN. An RNN reads text one word at a time and carries a running summary forward. Google's 2017 blog post on the Transformer names two problems with that. Ties between distant words are hard to learn, because the number of steps between them grows with the distance. And reading one word at a time makes it hard to use the parallel hardware, GPUs and TPUs, that training runs on.

A transformer removes the one-word-at-a-time step. It processes all the words in parallel.

How attention works

Attention is how a word gets context from the words around it. Google's post uses this sentence: "I arrived at the bank after crossing the river." The word "bank" could mean a building or a riverbank. Only "river" settles it.

The Transformer handles that in four moves, as the post describes them:

  1. It compares the representation of "bank" with every other word in the sentence.
  2. Each comparison produces an attention score, which says how much that word should contribute.
  3. The scores weight an average of all the words' representations.
  4. That average passes through a small fully connected network and becomes a new representation of "bank". Because "river" scored high, the new version leans toward riverbank.

The paper writes this as one formula: Attention(Q, K, V) = softmax(QKᵀ / √d_k)V. Q, K and V stand for queries, keys and values, three versions of each word's representation. In self-attention, the paper says, "all of the keys, values and queries come from the same place." That means the words in a sentence attend to each other, not to something outside it.

The paper also uses multi-head attention. It runs several attention operations side by side, each on its own learned projection of the same input, then joins the results. The base design uses 8 heads of 64 dimensions each, which lets the model track different kinds of relationships at once.

What the original design looked like

The 2017 model had an encoder and a decoder. The encoder reads the input sentence. The decoder writes the output one word at a time.

  • The encoder is a stack of 6 identical layers. Each has a self-attention step and a feed-forward step, with residual connections and layer normalization around them.
  • The decoder is also 6 layers. It adds a third step that attends to the encoder's output. Its own self-attention is masked, so a prediction can depend only on words that came earlier.
  • Every layer works with vectors of 512 numbers. The feed-forward step widens them to 2,048 inside.

Attention alone has no sense of word order, since it compares all words at once. The paper fixes this by adding a position signal to each word's embedding, built from sine and cosine waves of different frequencies. The authors also tried learned position embeddings and got nearly the same translation score: 25.7 BLEU against 25.8 for the base model. They kept the sine version because it might handle longer sequences than the model saw in training.

A word on terms. The units the model reads are tokens, pieces of words, and the what are tokens in AI page explains how text is split. How many tokens a model can attend to at once is its context window, covered in what is a context window.

What the paper showed

The authors tested on machine translation, scored with BLEU, where higher is better. All numbers below are from the paper, on the WMT 2014 benchmarks.

Model English to German English to French
Transformer (base) 27.3 38.1
Transformer (big) 28.4 41.8

The abstract says the 28.4 on English to German beats the previous best results, including ensembles, by more than 2 BLEU. The big model trained for about 3.5 days on eight NVIDIA P100 GPUs. The paper calls that a small fraction of the training cost of the best models in the literature. The base model trained in about 12 hours.

The paper also tried the model on English constituency parsing, a different task, and reports that it generalizes there. Google's post adds that the English-to-German model, with little adaptation, beat all but one earlier approach on parsing.

These are the authors' results on one task family, on a benchmark that is now nearly a decade old. They show the design worked well and trained fast in 2017. They say nothing about any specific model sold today.

The limits the paper points to

The cost of self-attention grows with the square of the sequence length. The paper gives it as O(n²·d), where n is the number of tokens. Every word compares with every other word, so doubling the text roughly quadruples that part of the work. This is one reason context windows are expensive to extend.

The authors name their own next steps: using the model on inputs and outputs that are not text, such as images, audio and video, trying attention limited to a neighborhood of nearby positions, and making generation less sequential. Output is still written one token at a time.

Later work has changed the recipe in many ways, including designs that run only part of the network for each token. The what is mixture of experts page covers one of them. The core step, every token attending to every other token, is the part that came from this paper.

Why it matters

The change was a move from sequential to parallel. Once the model no longer had to read word by word, training could use more hardware at once, and the design became a base that others kept building on. That is why the paper's title, which sounded like a bet in 2017, now reads as a description.

If you want to read the source, the arXiv page has the abstract and PDF, and an HTML version is at arxiv.org/html/1706.03762. The authors point to the open-source Tensor2Tensor library as their code.

Frequently asked questions

What is a transformer model in AI?

It is a neural network design that relies on attention instead of recurrence or convolutions. Every word in a text is compared with every other word at once, so the model can use context from anywhere in the input.

Who invented the transformer?

Eight authors listed on the 2017 paper 'Attention Is All You Need': Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser and Illia Polosukhin. Google Research published a write-up of it on August 31, 2017.

What is self-attention in a transformer?

Self-attention is attention where the queries, keys and values all come from the same text. Each word scores every other word in the sentence and takes a weighted average of their information.

What are the limits of the original transformer?

The paper gives the cost of self-attention as O(n²·d) in sequence length n, so long inputs get expensive. Generation also still produces output one word at a time.

Sources

  1. Attention Is All You NeedarXiv
  2. Attention Is All You Need (HTML version)arXiv
  3. Transformer: A novel neural network architecture for language understandingGoogle Research

Add ClaudeGPT Si as a preferred source on Google