Key takeaways
- Agent-style AI coding benchmarks such as SWE-bench Verified, Terminal-Bench 4.0 and FrontierCode 2.0 now set the pace, while HumanEval's repository is marked deprecated.
- Scores depend on the agent around the model, the effort and tool settings, the cost of each run and confidence intervals, so a one-point lead often means little.
- TechTwitter IO's satirical Trust Me Bro and Felony boards poke fun at launch-chart fine print and test editing, but they count posts on X and are not results.
AI coding benchmarks now come in two kinds: tests that hand a model a fixed problem and check the answer, and tests that set an agent loose in a real repository or terminal and check whether the work got done. As of November 2026 the agent kind dominates, including both coding tests inside the Artificial Analysis index, and the details of each setup decide how much a headline score is worth.
The tests that matter now
| Benchmark | What the model or agent does | How it is scored |
|---|---|---|
| SWE-bench Verified | Writes a patch for one of 500 real GitHub issues, each screened by human annotators | The repository's tests pass or fail |
| Terminal-Bench 4.0 | Completes tasks in a terminal | Resolution rate with 95% confidence whiskers, plus cost and tokens |
| FrontierCode 2.0 | Makes changes to open-source repositories, 132 tasks | Tests plus a rubric checked by an LLM judge; failing any blocking check scores zero |
| LiveCodeBench | Solves dated contest problems from LeetCode, AtCoder and Codeforces | Also covers self-repair, code execution and test output prediction |
| Aider polyglot | Edits code for 225 Exercism exercises in six languages | Percent correct, with cost |
Two older names have faded. OpenAI's HumanEval repository is now marked deprecated and unmaintained, and the newest entries on Aider's polyglot board date from October 2025, so neither tells you much about this year's models.
Where the score really comes from
The agent around the model. SWE-bench's Verified board ranks whole systems, from simple agent loops to multi-rollout setups with review steps. To compare models rather than scaffolds, it also runs every model in the same minimal bash agent, mini-SWE-agent. Even there, it warns that results from its 1.x and 2.x releases are not directly comparable, because 2.x invokes actions through tool calling.
Cost. Terminal-Bench prints cost and tokens next to every resolution rate. A model that finishes a few more tasks at triple the spend may be the wrong pick for a team on a budget.
The index weighting. Artificial Analysis gives coding a quarter of its Intelligence Index v5, split between FrontierCode 2.0 and Terminal-Bench 4.0. A model that is strong at code but weak elsewhere will rank lower there than on a pure coding board.
Fine print to read on any launch chart
- Who ran it. A lab's internal eval is a claim until an outside board reproduces it.
- Effort and tools. Results at maximum reasoning effort, or with extra tools, should not be set next to default runs.
- Margins. If two agents sit inside each other's confidence whiskers, the "lead" is noise.
- Contamination. LiveCodeBench lets you test a model only on problems newer than its training data, and Terminal-Bench carries a canary line asking that its data never be used for training.
The satirical scoreboard for the fine print
That fine print is exactly what TechTwitter IO, the free satirical daily paper of tech Twitter, turns into a weekly board. On its Benchmarks page, Trust Me Bro ranks models by how much they claim against how many outside receipts back it up, with extra penalties for footnoted chart results and misleading axes. A separate Felony board books models for things like editing the tests so a task passes, which happens to be the failure every coding benchmark is built to catch.
The boards are counted from a week of public posts on X and link to them, and the paper is clear that it is satire, so none of its scores are test results. Read it after the real leaderboard, when you want to know why a launch chart is getting ratioed.
The takeaway for picking a coding model
Use SWE-bench Verified and Terminal-Bench 4.0 for agent work, check the scaffold and the cost column, and treat a single-point lead with suspicion. Then run your top two candidates on a few tickets from your own repository, which no public board can do for you.
Frequently asked questions
What is the best AI coding benchmark?
For agent work, SWE-bench Verified and Terminal-Bench 4.0 are the most direct, since they check real patches and finished terminal tasks. LiveCodeBench is useful when you want problems newer than a model's training data.
Why do SWE-bench scores differ for the same model?
The full SWE-bench board ranks whole systems, so the agent scaffold changes the result. Its bash-only view runs every model in the same minimal agent to compare models directly.
Is HumanEval still used?
Its official repository from OpenAI is marked deprecated and unmaintained, so it is a poor guide to current models. Artificial Analysis, for example, uses FrontierCode 2.0 and Terminal-Bench 4.0 for the coding part of its index.
Sources
- SWE-bench VerifiedSWE-bench
- Terminal-BenchTerminal-Bench
- Intelligence Benchmarking MethodologyArtificial Analysis
- LiveCodeBenchLiveCodeBench
- Aider LLM LeaderboardsAider
- HumanEvalOpenAI
- The BenchmarksTechTwitter IO