Anthropic study: experts follow an AI agent's lean 86% of the time

Anthropic published a study on October 27 showing that experts follow an AI agent's lean more often than not. In the experiment, 50 professionals matched a randomly assigned "lean" from their Claude Code agent on 86% of analytical decisions, where chance would give about 50%.

The finding matters because the analysts did not know the lean was random. Anthropic argues that when an expert signs off on an agent's work, the sign-off may not reflect independent judgment.

How the experiment worked

Anthropic recruited 61 professional analysts with causal-inference experience who regularly use Python or R. Eleven declined to use the agent and were analyzed separately. The other 50 advised a fictional farming cooperative on whether to expand a paid advisory program.

The task had four open decisions, each with two defensible options:

• Which outcome to measure: yield per hectare or yield per unit of operating cost.
• Which comparison method to use: difference-in-differences or a covariate-adjusted comparison.
• Which farms to cover: enrolled farms only or the full membership.
• How to weight farms: equally or by producer size.

A coin flip secretly set which option each agent would favor on each decision. The agents ran Claude Opus 4.8 inside Claude Code and were told to argue their lean honestly without misleading anyone. Analysts were not told about the leans. The median task took 65 minutes.

Experts follow the AI agent's lean 86% of the time

Across the four analytical decisions, analysts picked the option the agent leaned toward 86% of the time. By decision, that was 92% for the outcome, 82% for the comparison method, 81% for coverage and 90% for weighting.

The final recommendation moved too, though less. Analysts recommended rolling out the program 54% of the time when the agent leaned that way, against 36% otherwise.

Among 38 cases where an analyst already had a preference before the agent weighed in, they switched 60% of the time when the agent disagreed. They switched 6% of the time when it agreed. Anthropic notes the confidence interval on that 54-point gap is wide, from 24 to 82 points.

Few analysts said the agent influenced them

Of the analysts, 77% repeated the agent's arguments in at least one explanation, sometimes word for word. Only 15% credited the agent. Fifty-two percent said the recommendation was more their own than the agent's.

Six of 48 respondents guessed the study was about AI influence, and none suspected the agent's preferences had been manipulated. Experience, self-rated expertise, education, quiz score and trust in AI did not predict who deferred.

The agent did help in one way. Six of the 11 analysts who worked without it (55%) made serious methodological errors, against 3 of 50 (6%) with it. Anthropic cautions that the groups were not comparable, because the group without AI was less experienced.

What the study can't tell us

Anthropic lists its own limits:

• The leans were deliberately strengthened to measure their effect. Real agents may lean less, so the effect in practice could be smaller.
• It covered one task, one field (farming, which the analysts were new to) and one model. Real experts would likely hold stronger views and could ask stakeholders.
• The results are averages. Because the leans were random, the study can't measure how much any one analyst was influenced.

With no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so its defaults were fairly stable.

Why Anthropic thinks it matters

Anthropic's concern is scale. If many research teams defer to models with similar preferences, agreement among them may not be independent evidence, and one arbitrary preference could bias many results at once. It also says the finding bears on AI safety work, where human review is meant to be an independent check.

The company says this is not an argument against using AI. It suggests measuring deferral in everyday work, building agents that present the best case for each option or ask for the expert's view first, and deciding on purpose which judgments need a human. The paper is on Anthropic's research page. Related coverage: Anthropic buys Mithril Medicine and details Claude drug work.