Paper: AI benchmarks overweight some difficulty levels, skip others

A new paper argues that every AI benchmark has a hidden shape, its benchmark difficulty distribution, and that the shape can quietly change who wins a leaderboard. The authors, led by Yifan Sun, audited 25 popular benchmarks and found that many of them overweight some difficulty levels and leave others empty. The paper, "Your Benchmark Is an Empirical Measure Over Difficulty", was posted to arXiv on October 21, 2026, is accepted at NeurIPS 2026, and was the most upvoted paper on Hugging Face's daily papers page on October 30.

It matters because leaderboard scores drive real decisions: which model a team picks, and whether a training run is judged to have worked. If the test is lopsided, those decisions rest on a number that doesn't mean what people think it means.

The idea: a benchmark is a histogram of difficulty

Take a test with 500 questions. Some are easy, some are hard. Line them up by difficulty and you get a histogram. The paper says a benchmark score is just model performance averaged over that histogram, so the histogram decides what the score measures.

That gives two ways for a benchmark to fail.

Measure mismatch. The benchmark's difficulty mix differs from the mix you care about. Two models can tie on the benchmark and still differ a lot on the hard questions you actually need them to solve. The paper's teaser figure shows two models scoring 72.1 and 72.4 on a benchmark, but 61.8 and 68.9 on a target spread of difficulties.

Coverage gaps. There is a stretch of difficulty with few or no questions. If training really improves a model in that stretch, the benchmark barely moves. In the paper's illustration, a score goes from 64.2 to 64.6 while real progress is missed.

How they measured it

The team estimated each question's difficulty with a 1PL item response model, a standard statistical tool from educational testing. It rates a question as hard if strong models fail it, rather than just counting failures. For non-agentic benchmarks they used 12 language models from 1.7B to 397B parameters, with up to 32 attempts per question. For agentic benchmarks they used published results from the HAL leaderboard covering 15 to 66 closed-source models.

They defined two scores, both lower-is-better:

• Difficulty Distribution Mismatch (DDM) measures how far the benchmark's difficulty spread is from an even spread across easy to hard. The authors call an even spread a neutral default when you know nothing else, not the one correct target.
• Max Gap is the widest empty stretch between neighboring questions on the difficulty scale.

What they found

Across 25 benchmarks in seven domains (math, coding, agentic tasks, logic, science, instruction following and medicine):

• 10 of 25 have a DDM of 0.10 or higher. The worst are ScienceQA (0.178), MATH-500 (0.164), MBPP+ (0.163), IFEval (0.159) and MedQA (0.157).
• 8 of 25 have a Max Gap of 0.10 or higher. The largest are AMC23 (0.217), AIME24 (0.179), HumanEval (0.165), AIME25 (0.158) and SWE-bench-Mini (0.149).

The authors note that gaps are not only a small-benchmark problem. HumanEval has 164 problems and BBH with five objects also shows large gaps. They also admit that Max Gap naturally grows as a benchmark gets smaller, but argue an uncovered stretch is a real limitation either way.

Their reading is cautious: a high mismatch means a leaderboard result reflects the benchmark's own difficulty weighting, so rankings should be read with care. They do not claim any specific ranking is wrong.

BenchFill: writing the missing questions

The team also built BenchFill, a method that fills gaps. A language model (MiniMax-M2.7) rewrites existing questions into variants aimed at the middle of an empty stretch. Each new question is rated for difficulty on the original scale, and a separate model (GPT-5.4) checks that its answer is correct. Human experts reviewed the final items.

On eight benchmarks, adding three new questions each cut the sum of the three largest gaps by 21% to 39%, averaging 32.7%. An idealized oracle that picks the best possible questions averaged 35.1%, so the method gets close to the ceiling the authors computed. On AIME24, a baseline that inserted questions without targeting a gap cut it by 6.9%, against 35.6% for BenchFill.

They released expanded versions of four math benchmarks on Hugging Face: AMC23 (40 to 80 questions), AIME24 (30 to 50), AIME25 (30 to 52) and AIME26 (30 to 53), after manual review removed weak or wrong items.

Limits to keep in mind

The method audits benchmarks after the fact. It does not yet tell you how to build a good one from scratch, which the authors list as future work. Difficulty estimates depend on the item response model and on which models were sampled, and the agentic results come from a different, third-party data source. BenchFill still needs human review and leans on an LLM judge for validity. And the paper's finding is about the shape of the test, not proof that any lab's reported number is wrong.

Why it matters

This sits next to other recent work on shaky evaluation, such as Ai2's finding that LLM novelty judges flip verdicts on small prompt changes, and next to index builders who combine many tests, like the Artificial Analysis Intelligence Index. The practical takeaway for anyone choosing a model: a single benchmark score is an average over whatever questions the authors happened to include. Check what the questions look like before trusting a two-point gap.

The difficulty estimates for all 25 benchmarks are in the authors' GitHub repository, the expanded math sets are on Hugging Face, and the project page is at benchmark-difficulty-measure.github.io.