Key takeaways

  • Attention ignores word order on its own, so transformers add positional encoding to tell tokens apart by position.
  • RoPE rotates query and key vectors so attention scores depend on the distance between tokens, and LLaMA models use it.
  • Position Interpolation and YaRN stretch RoPE models past their training length, with YaRN taking Llama 2 to 128,000 tokens.

Positional encoding is the signal that tells a language model where each word sits in a sentence. Without it, a transformer would read "dog bites man" and "man bites dog" as the same bag of words. Many open models, including Meta's LLaMA family, get that signal from rotary position embeddings, or RoPE, and the way RoPE works also decides how far a model's context window can be stretched.

The topic is back in front of developers because Sebastian Raschka published "Understanding Positional Embeddings in Modern LLMs" on November 15, 2026, and it reached the Hacker News front page with 138 points that day. This explainer goes back to the papers behind each method and what they measured.

Why a transformer needs positional encoding

The core of a transformer is attention: every token (a word or piece of a word) scores how relevant every other token is, then mixes in their information. That score is a dot product between two vectors, one for each token. Nothing in it depends on where the tokens are. Shuffle the input and each token gets the same scores from the same neighbors.

So position has to be added from outside. Four main methods do it.

Sinusoidal encoding: a fingerprint added to each word

The original transformer paper, Attention Is All You Need (2017), added a fixed vector to each token's embedding before the first layer. The vector is built from sine and cosine waves of different speeds. Some dimensions cycle quickly, others slowly, so each position gets its own pattern, like the hands of a clock read together.

A small example with four dimensions. Two of them use a fast wave (one radian per position), two use a slow one (a hundredth of a radian):

Position sin fast cos fast sin slow cos slow
1 0.84 0.54 0.01 1.00
2 0.91 -0.42 0.02 1.00

Fast waves tell neighbors apart, and slow waves keep counting after the fast ones repeat. The real model used 512 dimensions.

The authors picked sine waves because the encoding of position 10 + k is a fixed linear function of the encoding of position 10, which they hoped would let the model learn relative distances. They also tried learned position vectors and got nearly the same translation score: 25.7 BLEU against 25.8. They kept the sine version because it "may allow the model to extrapolate" to longer inputs than it was trained on. Later work showed it mostly doesn't.

RoPE: rotate instead of add

RoFormer, by Jianlin Su and colleagues (first posted April 2021), introduced rotary position embedding. Instead of adding a vector to the word, RoPE rotates the query and key vectors inside attention. Each pair of dimensions turns by an angle equal to the position times a fixed frequency, with fast and slow frequencies chosen the same way as in the sine scheme.

The worked example makes the point. Say one pair of dimensions turns 10 degrees per position. A query at position 5 is turned 50 degrees. A key at position 3 is turned 30 degrees. Their dot product only sees the 20-degree gap between them. Move both to positions 105 and 103 and the gap is still 20 degrees, so the score is the same. The model sees how far apart two words are, not where they sit in absolute terms.

The paper also shows that, with these frequencies, attention tends to fade as tokens get farther apart. Its results were modest: 27.5 BLEU against 27.3 for a base transformer on English-German translation, and 69.79% against 68.10% for WoBERT on a Chinese long-text matching task when RoFormer read 1,024 tokens.

The approach caught on anyway. Meta's LLaMA uses RoPE, according to the Position Interpolation paper discussed below, and the YaRN paper builds on Llama 2 for the same reason.

ALiBi: no position vectors at all

ALiBi, from Ofir Press, Noah A. Smith and Mike Lewis (August 2021), skips position vectors entirely. It subtracts a penalty from each attention score in proportion to the distance between the two tokens. Each attention head gets its own fixed slope; with eight heads they run from 1/2 down to 1/256.

The paper's headline result: a 1.3-billion-parameter model trained on 1,024-token inputs matched a sinusoidal model trained on 2,048 tokens, at 8.92 against 9.01 perplexity, while training about 11% faster. The sine version fell apart past its training length. Trained on 512 tokens, its perplexity rose from 20.05 to 178.97 at 3,512 tokens. Rotary degraded more slowly, to 43.14 at 5,512.

The authors flag a catch. ALiBi's gains on longer inputs may come mostly from fewer tokens being predicted with too little context, not from the model actually using the extra text.

NoPE: let the causal mask do it

The Impact of Positional Encoding on Length Generalization in Transformers (May 2023) tested a blunt idea: no positional encoding at all, called NoPE. In a decoder model, each token can only attend to tokens before it. The authors prove this mask alone lets the model work out absolute and relative position.

They trained models of about 107 million parameters on ten synthetic tasks such as copying, addition and sorting, at lengths up to 20, then tested up to 40. NoPE matched or beat every explicit method. Sinusoidal and rotary did worst. The limit is scale. These were small models on toy tasks, and the authors say their 1-billion-parameter follow-up runs are preliminary.

Stretching RoPE to longer contexts

Since many open models use RoPE, the practical question became how to run them past the length they were trained on. Feeding in positions the model never saw breaks it: Position Interpolation (Meta, June 2023) reports perplexity above 1,000 when LLaMA goes past its 2,048-token window without changes.

The fix in that paper is simple. To run at 8,192 tokens, divide every position by four, so position 8,000 is treated as 2,000. About 1,000 fine-tuning steps were enough to extend LLaMA models up to 32,768 tokens. Plain fine-tuning without the trick moved the usable window only from 2,048 to 2,560 tokens after 10,000 steps. The cost: scores within the original window dropped slightly, with BoolQ falling from 76.1 to 73.2 for the 7B model.

YaRN (August 2023) refined it. Squeezing every frequency equally blurs the fast-turning dimensions that track nearby words, so YaRN leaves those alone, squeezes only the slow ones, and adds a small adjustment to attention. The authors extended Llama 2 7B and 13B to 128,000 tokens, with 99.4% accuracy on a test that hides a passkey in long text, and claim it needs 10 times fewer tokens and 2.5 times fewer training steps than earlier methods.

What this means for people using LLMs

The context window on a model's spec sheet is partly a statement about its positional encoding. A long window is often a RoPE model trained short and stretched later. The papers show stretching works, but each one also records a small loss on short tasks, and the YaRN authors note that low perplexity is not proof a model can find facts deep in a long prompt. When a vendor quotes a long context, a retrieval test at that length tells you more than the number.

Frequently asked questions

What is the difference between positional encoding and positional embeddings?

The terms are used almost interchangeably. Positional encoding often means a fixed formula such as the original sine waves, while positional embeddings often means vectors the model learns, but both give the model position information.

What is rotary position embedding (RoPE)?

RoPE, introduced in the RoFormer paper in 2021, rotates the query and key vectors in attention by an angle tied to each token's position. The attention score then depends only on how far apart two tokens are.

Why did the original transformer use sine and cosine?

The authors hoped sine waves would make relative distances easy to learn and let the model handle longer inputs. Learned position vectors scored almost the same, 25.7 against 25.8 BLEU.

Can a transformer work without positional encoding?

A 2023 study found that decoder models with no positional encoding (NoPE) can infer position from the causal mask and generalized to longer inputs better than explicit methods on small synthetic tasks.

Sources

  1. Attention Is All You NeedarXiv
  2. RoFormer: Enhanced Transformer with Rotary Position EmbeddingarXiv
  3. Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationarXiv
  4. The Impact of Positional Encoding on Length Generalization in TransformersarXiv
  5. Extending Context Window of Large Language Models via Positional InterpolationarXiv
  6. YaRN: Efficient Context Window Extension of Large Language ModelsarXiv
  7. Understanding Positional Embeddings in Modern LLMsAhead of AI (Sebastian Raschka)

Add ClaudeGPT Si as a preferred source on Google