Key takeaways

  • Perplexity measures how well a language model predicts unseen text, and a lower score means better predictions.
  • It is the exponential of the average negative log-likelihood per token, which is the model's cross-entropy loss.
  • Scores only compare on the same test text, tokenizer and evaluation method, which is the main weakness of OpenAI's November 22, 2026 proposal to use it across labs.

Perplexity in AI is a score for how well a language model predicts what comes next in text it has never seen: the lower the number, the less "surprised" the model is by each next word, and the better it predicts. This is the evaluation metric researchers use on language models, not Perplexity, the AI search company with the same name.

The metric is in the news because OpenAI published the perplexity of its own models on November 22, 2026, in a post called Perplexity and Intelligence, and proposed that AI labs agree on limits to it. The proposal itself is covered in OpenAI publishes model perplexity, urges shared limits.

What perplexity measures in language models

A language model reads text one token at a time. A token is a word or part of a word. Before each token, the model gives a probability to every possible next token. Perplexity asks one question of those guesses: how much probability did the model give to the token that actually came next?

High probability on the real next tokens means low perplexity. Thin or wrong bets mean high perplexity. The textbook Speech and Language Processing by Daniel Jurafsky and James H. Martin puts it plainly: "the lower the perplexity of a model on the data, the better the model."

It is always measured on test text the model did not train on. Otherwise, the textbook warns, the score comes out artificially low.

How perplexity is calculated

Hugging Face's documentation defines perplexity as "the exponentiated average negative log-likelihood of a sequence." In steps:

  1. For each token in the test text, take the probability the model gave it, given all the tokens before it.
  2. Take the logarithm of each probability and flip the sign. This is the negative log-likelihood: small when the model was confident and right, large when it was surprised.
  3. Average those numbers over all the tokens.
  4. Raise e to the power of that average.

The average in step 3 is the cross-entropy loss, the same number a model minimizes while it trains. So perplexity is the exponential of the loss, and the log of perplexity is the loss itself. Hugging Face notes the same equivalence: perplexity is the exponential of the cross-entropy between the data and the model's predictions. In its own code example, the last line is simply ppl = torch.exp(avg_nll).

Perplexity as the number of choices a model hesitates between

The easiest way to feel the number is the "branching factor" reading in Jurafsky and Martin. Take a toy language with three words, red, blue and green. A model that thinks each word is equally likely has a perplexity of exactly 3. It is as confused as if it had to pick among three equal options at every step.

Now train a second model that has learned red is common: it gives red a probability of 0.8 and the other two 0.1 each. On the test text "red red red red blue", its perplexity falls to 1.89, even though there are still three possible words. Perplexity is the effective number of choices the model is hesitating between. A perplexity of 1 would mean it always knew the next token for certain.

What is a good perplexity score

There is no single good perplexity score. The number depends on the test text, the tokenizer and the setup, so it only means something next to other numbers measured the same way. Some reference points from primary sources:

Model and test Perplexity Source
Unigram model, Wall Street Journal test set 962 Jurafsky and Martin
Bigram model, same test set 170 Jurafsky and Martin
Trigram model, same test set 109 Jurafsky and Martin
GPT-2 large on WikiText-2, as reported in the GPT-2 paper 19.93 Hugging Face docs
GPT-2 large on WikiText-2, chunks with no overlap 19.44 Hugging Face docs
GPT-2 large on WikiText-2, sliding window with stride 512 16.44 Hugging Face docs

Each n-gram model that looks at more context scores lower. The GPT-2 rows show how much the method alone moves the number: the same model on the same text scores 19.44 or 16.44 depending only on how much earlier text it sees for each prediction.

GPT-2 large reads at most 1,024 tokens at once, so a long test text has to be cut up. Cutting it into separate chunks starts every chunk with no context and gives a worse score. Hugging Face recommends a sliding window, which moves the context forward in steps (the stride) so each token has more text behind it.

Why you can't compare perplexity across models

Perplexity looks like a universal number, but two scores are only comparable under strict conditions:

  • Same tokenizer. Hugging Face warns that the way text is split into tokens directly changes a model's perplexity. A model that splits text into more, smaller tokens has an easier guess at each step. Jurafsky and Martin say two models' perplexities are only comparable if they use identical vocabularies.
  • Same test text. A model scores lower on text that looks like its training data.
  • Same evaluation method. Context length and stride change the result, as the GPT-2 table shows.
  • The right kind of model. Hugging Face notes the metric applies to models that predict the next token, such as GPT-style models, and is not well defined for masked models like BERT, which fill in blanks.

Even then, Jurafsky and Martin write that a better perplexity is no guarantee of better results on a real task, so gains should be confirmed with tests on the task itself.

Why AI labs still track perplexity

The 2020 paper Scaling Laws for Neural Language Models by Jared Kaplan and colleagues found that a model's cross-entropy loss, the log of its perplexity, falls as a power law as model size, data and training compute grow. That predictability is why labs watch the loss curve while they train.

OpenAI's November 22 post goes further and argues that this number tracks intelligence. Chief scientist Jakub Pachocki reports log-perplexity, which is cross-entropy loss, on a fixed set of held-out data from OpenAI's internal codebase:

Model Log-perplexity, per OpenAI
GPT-3.5 about 0.99
GPT-4 about 0.91
Current models below 0.8
An unnamed upgraded model about 0.76

The post ties each drop to new abilities, and says OpenAI expects further big jumps in ability somewhere around 0.68 to 0.7. The post doesn't name the base of its logarithm. If it is the natural log used in Hugging Face's recipe, 0.99 is a perplexity of about 2.7 and 0.76 about 2.1: on OpenAI's code test, the newer model hesitates between roughly two options per token instead of nearly three.

OpenAI lists the weak points itself: labs test on different data, perplexity maps onto no single task, and it misses gains made after pretraining, such as reasoning at run time.

What to watch

OpenAI's proposal only works if other labs measure the same way, so the next sign is whether Anthropic, Google DeepMind or Meta publish comparable numbers. Anyone reading them should apply the textbook rule: compare perplexity only on the same text, tokenizer and method.

Frequently asked questions

What is a good perplexity score?

There is no universal good score, because perplexity depends on the test text, the tokenizer and the method. On the same Wall Street Journal test, a trigram model scored 109 against 962 for a unigram model, and GPT-2 large scores about 16 to 19 on WikiText-2.

Is lower or higher perplexity better?

Lower is better. A lower perplexity means the model gave more probability to the words that actually came next, and a perplexity of 1 would mean it predicted every token with certainty.

What is the perplexity score formula?

Perplexity is e raised to the average negative log-likelihood of the tokens in a test text. Put another way, it is the exponential of the cross-entropy loss, so the log of perplexity is the loss itself.

Can perplexity be used for BERT?

Not in its usual form. Hugging Face's documentation says perplexity applies to models that predict the next token and is not well defined for masked language models like BERT.

Sources

  1. Perplexity of fixed-length modelsHugging Face
  2. Speech and Language Processing, chapter 3: N-gram language models (draft of August 19, 2026)Daniel Jurafsky and James H. Martin, Stanford University
  3. Perplexity and IntelligenceOpenAI
  4. Scaling Laws for Neural Language ModelsarXiv
  5. OpenAI publishes model perplexity, urges shared limitsClaudeGPT Si