Key takeaways
- Ai2 says GPT-5.4 scored 93.5% on one prompt and 40.9% on a simpler one when picking the more novel of two ideas.
- Retrieval and larger reasoning budgets gave little reliable benefit across six tested judges.
- A cheap Claude Opus 4.6 prompt with majority voting scored 0.854 macro-F1 for $2.59, beating two purpose-built novelty evaluators.
LLM novelty judges are unreliable, according to a new study from the Allen Institute for AI (Ai2). In a blog post published on October 27 and a paper on arXiv, the team shows that language models asked to decide whether a research idea is new can flip their answers when the prompt changes slightly, sometimes ending up worse than a coin toss.
This matters because labs are building AI systems that generate research ideas, and those systems need something to score the ideas. If the scorer is unstable, the scores say little about whether an idea is actually new.
What a novelty judge is
A novelty judge is an LLM that reads a research idea and says whether it is new compared with prior work. It can rate one idea on its own (pointwise) or pick the more novel of two (pairwise). Automated research agents use judges like this to filter and rank what they produce.
The hard part is knowing the right answer. Ai2 did not use paper acceptance as a stand-in for novelty. Instead, the team mined ICLR 2026 peer reviews for explicit comments on originality, and built two pools:
- 154 ideas from accepted papers in the top 10% of their area, where reviewers praised novelty and none disputed it.
- 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none praised novelty.
In a check of the review mining, an annotator agreed with the extracted label in 47 of 50 cases, according to Ai2.
The team then built two test settings. In Human-Only, judges try to separate the high-novelty pool from the lower-novelty one. In Human+Generated, the 154 validated human ideas are paired with 154 ideas from a simple LLM generator that got only a research area and a request for a novel idea. The generated ideas are assumed to be less novel. That is a weak label, not a verified one. An expert reviewed 30 pairs, weighted toward cases the strongest judge got wrong, and preferred the human idea in all 30.
Ai2 tested six judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.
Small prompt changes flip the verdict
The headline result is about prompts. For GPT-5.4 on Human+Generated pairs, a prompt telling the judge that reviewers found exactly one of the two ideas novel produced 93.5% accuracy, according to Ai2. A simpler prompt asking which idea was novel produced 40.9%, below the 50% a random guess would give. The model changed its verdict on more than half of the pairs.
The paper's abstract says pairwise accuracy can shift by more than 50 points from prompt changes alone. Human-Only comparisons were steadier but not immune: the same change moved accuracy by about 7 points, and Claude Opus 4.6 still shifted by 21 points.
There is also no prompt that works for everyone. Removing an instruction that warns against counting trivial or incremental work as novel helped some judges and hurt others. Swapping which model generated the ideas changed roughly a quarter to a third of verdicts on average across the six judges.
More search and reasoning did not fix it
A common answer is to give the judge related papers or more thinking time. Ai2 found little reliable benefit from either.
- An agentic judge with extra-high reasoning and live arXiv search scored 0.78 macro-F1, about 11 points behind Claude Opus 4.6 with five pre-retrieved abstracts, which scored 0.89. Ai2 says the agentic judge cost roughly 10 times more per decision.
- A simple Claude Opus 4.6 prompt with low reasoning and a majority vote over three calls scored 0.854 macro-F1 for a total of $2.59.
- Two purpose-built novelty evaluators did worse: Idea Novelty Checker scored 0.642 at $30.66, and the novelty evaluator from Sakana AI's AI Scientist scored 0.772 at $82.05.
In other words, the cheapest prompted baseline beat both dedicated systems on accuracy and cost.
Limits of the study
The authors list several. Labels for the generated ideas are assumed, and only 30 pairs got expert review. The data comes from ICLR 2026 reviews, so it may not carry over to other fields or venues. Only one simple generator produced the AI ideas, and a stronger generator that searches and iterates could behave differently. The review mining has some noise.
The authors do not claim to have a fixed fix. They suggest testing judges on the kind of ideas they will actually score, measuring how much verdicts move across prompts and models, and improving how prior literature is brought in.
What to watch
Ai2 released the benchmark, data on Hugging Face and code on GitHub, with a pipeline others can run on new peer reviews and other scientific communities. The practical advice for anyone using an LLM to score ideas: run several prompt variants, check how often the verdict flips, and do not treat one score as a measurement.
Frequently asked questions
Can LLMs judge whether a research idea is novel?
Not reliably, according to Ai2. Across six judges, small prompt changes shifted pairwise accuracy by more than 50 points, and verdicts flipped on more than half of identical pairs in one case.
Does giving an LLM judge search access help it assess novelty?
Only a little. Ai2 found retrieval and bigger reasoning budgets gave small or inconsistent gains, and an agentic judge with live arXiv search scored about 11 points below Claude Opus 4.6 with five pre-retrieved abstracts.
How did Ai2 decide which research ideas were novel?
It mined ICLR 2026 peer reviews for explicit comments on originality, keeping ideas from the top and bottom 10% of their area where reviewers agreed. That gave 154 higher-novelty and 145 lower-novelty human ideas.
