AI agent harness benchmark finds 31% of passing runs break rules

A new benchmark for the AI agent harness, the software that wraps a model and lets it act, finds that a harness can fail even when the agent finishes the job. In "Trial by Trace," posted to arXiv on October 23, 2026, a team at TraceLite AI built TraceTrial-Bench to score harnesses separately from models. Across 3,420 runs, 31.1% succeeded at the task while still failing at least one harness criterion.

That matters if you pick or build coding agents. Most leaderboards report whether the task got done. They can't tell you whether the tool behind the model kept its context straight, respected its permissions or left an honest record of what it did.

What an agent harness is, and the problem

An agent is a model plus a harness. The harness feeds the model its context, runs the tools it calls, enforces limits, saves state between steps and logs what happened. Claude Code and Codex are harnesses. So are OpenClaw and DeepSeek's harness, the other two in the study.

The paper's point is that results mix two things. A run can fail because the model reasoned badly, or because the harness dropped context, ran a command it should have blocked or lost state. Benchmarks that only check the final outcome can't separate the two. They can also miss the reverse: a run that passes by luck while the harness did something unsafe along the way.

What the team built

TraceTrial-Bench has 95 tasks, each with a checkable answer, either an executable test or a check on the final state. The tasks are grouped by the part of the harness they stress:

Of the 95 tasks, 46 are single-turn, 45 are fixed multi-turn and 4 use a simulator that talks back to the agent.

Three modules do the measuring. Trajectory tracing converts each harness's native log into one shared format, without changing how the harness behaves. Evidence acquisition gathers proof for each criterion from the trace, the files the agent produced and the state of the environment. Failure attribution then links a failed harness criterion to a failed agent criterion through a mechanism chain written in advance.

Scoring has two layers, one for the agent's task result and one for the harness criteria. The overall score weights them 50/50. For criteria that need judgment, a model (deepseek-v4-flash) acts as the judge.

The results

The team ran four harnesses with nine models: DeepSeek-V4-Pro, Gemini 3.7 Flash, GLM-5.3, GPT-5.6-Sol, Grok-4.6, HY3, Kimi K3, Opus 5 and Qwen3.8-Max. That is 4 x 9 x 95 = 3,420 runs, one per combination. The paper's Table 6 gives, in percent:

Harness scores span 12.2 points, from 59.8% to 72.0%. Agent pass rates span 8.2 points.

The ranking changes depending on what you measure. Claude Code has the best task pass rate and overall score. OpenClaw has the best harness score and the most clean runs. So the harness that gets the most tasks done is not the one that behaves best underneath.

Across all runs, 39.5% passed every agent criterion, and only 8.4% passed every criterion in both layers. That 8.4% is the number to remember: fully clean runs are rare for every harness tested.

Limits to keep in mind

The authors list several themselves.

• Each model, harness and task combination was run once, so the paper doesn't show how much results shift between repeats.
• The judge model's output limit differed between evaluations, and the paper doesn't estimate how often the judge agrees with a human.
• A failed harness criterion is an operational shortfall, not proof of a bug. The attribution links are leads for review, not established causes.
• Coverage is uneven. Verification has a single task, and Execution has 27.
• The two layers are not fully independent, because one verdict depends on the agent's outcome.
• Run time differences are not a controlled speed comparison, since scheduling and recovery varied.

The benchmark and the paper come from the same team, and the harnesses were tested through adapters the team wrote. Independent reruns would add confidence in the exact scores.

Run it yourself

The code, tasks and configs are on GitHub under an MIT license. It needs Python 3.11 or later and Docker. You install it, add your model endpoint and key to a config file, build a harness image, then run one task or the full 95-task manifest. The project page links the leaderboard and a Hugging Face Space. The paper itself is on arXiv under CC BY-NC-SA 4.0.

If you compare coding agents, ask the question this paper raises: not only whether it finished, but what it did on the way.