Key takeaways
- OpenAI published log-perplexity scores for its models on November 22, 2026, from about 0.99 for GPT-3.5 to about 0.76 for an unnamed recent model.
- Jakub Pachocki proposed common limits on perplexity that labs would commit not to cross until a safety milestone is met.
- He wrote that a burst of pretraining advances could come as soon as 2027, and expects new abilities around 0.68 to 0.7.
OpenAI published the perplexity of its own models on November 22, 2026, and proposed that AI labs agree on limits to it. In a post titled Perplexity and Intelligence, chief scientist Jakub Pachocki argues that this one number tracks how capable a model is, and that the whole industry should measure it, publish it and commit not to push it past agreed points until safety research catches up.
Labs rarely share numbers about their pretraining runs, because those numbers tell rivals where they stand. OpenAI is now doing it on purpose, and tying it to a concrete proposal: a speed limit for the step that makes models smarter in the first place.
What OpenAI perplexity measures
Perplexity is a score for how well a model guesses the next token, the chunk of text it reads and writes. Lower is better. Pachocki reports it as log-perplexity, also called cross-entropy loss, measured on a fixed snapshot of OpenAI's internal codebase that the model never trained on. OpenAI has used that test set since GPT-3.5.
The post lines up the scores with what each model could do:
| Model | Log-perplexity | What it could do, per OpenAI |
|---|---|---|
| GPT-3.5 | about 0.99 | Fluent conversation, the original ChatGPT |
| GPT-4 | about 0.91 | Useful chain-of-thought reasoning |
| Models below 0.8 | under 0.8 | Large gains in cybersecurity and mathematics |
| An unnamed model | about 0.76 | Solutions to open math problems OpenAI had posed for over a year |
Pachocki says none of these abilities were clearly predictable in advance. They appeared as the score fell. The post also says loss dropped steeply in 2026, mostly from long-term research and more compute, with a smaller share from AI helping design its successors.
Why Pachocki points to 2027
The post compares pretraining research to mathematics, where AI went from little help to new results within about a year. Pachocki writes that "as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements." He frames that as a possibility, not a forecast.
He also says OpenAI expects "further transformative capabilities" once models reach a log-perplexity of about 0.68 to 0.7, and warns that checks on whether a model is aligned are getting harder as models learn to notice when they are being tested.
The proposal: limits labs agree not to cross
Pachocki proposes "common limits, based on perplexity and other measurements, that developers are committed to not crossing" until a set milestone is reached. His example of a milestone is a working successor to chain-of-thought monitoring, the practice of reading a model's reasoning to catch bad behavior. He also asks labs to share norms for evaluating models and to red-team safety cases before a model is first used.
He lists the weak points himself. Labs train on different data, so their numbers aren't directly comparable. Perplexity doesn't map to any single task. And it misses gains made after pretraining: reasoning at run time, covered in our explainer on test time compute, makes models better without lowering perplexity at all.
What to watch
The proposal only works if other labs join, and OpenAI's post names no other lab that has agreed to it. The timing matters: two days earlier METR said leading models are improving at research tasks faster than it can measure, which we covered in METR time horizon explained. The next sign will be whether Anthropic, Google DeepMind or Meta publish comparable numbers, or say why they won't.
