Key takeaways
- With identical tools and runtime, MCP and CLI interfaces changed Qwen3-14B's ALFWorld score from 56.7 (CLI) to 20.1 (MCP).
- The direction reversed on WebShop, where MCP scored 64.4 against 59.5 for CLI.
- The DART repair method raised Gemini-2.5-Flash scores in all six settings, by up to 9.0 points.
A new arXiv study asks whether an AI agent behaves the same when it reaches its tools through MCP or through a command line. The paper, by Zhenning Shi, Liang Pang and Yongqi Li, finds that it does not. With the same tools and the same runtime, MCP vs CLI changed scores by tens of points on some tasks.
MCP, the Model Context Protocol, is a standard way for an agent to call tools through structured calls. A CLI interface has the agent type shell-style commands instead. Many teams pick one by habit. This paper says the choice is part of how the agent decides.
What the researchers did
The authors built a test harness called MICA. It writes both interfaces from one shared description of the available actions, then runs every action through the same code. Only the way the model sees and calls the tools differs.
They ran six models: Qwen3 at 4B, 8B, 14B and 32B, DeepSeek-v4-Flash and Gemini-2.5-Flash. The tasks came from three standard environments: ALFWorld (household tasks), WebShop (shopping on a mock site) and ScienceWorld (multi-step science problems).
The results flip by model and task
No interface won everywhere. For Qwen3-14B on ALFWorld, MCP scored 20.1 and CLI scored 56.7. On WebShop the same model did the other way round: 64.4 for MCP against 59.5 for CLI. Token cost flipped too. MCP used 54.9k more tokens per ALFWorld task and 35.0k fewer on WebShop.
The authors also compared pairs of runs that started the same way. In 94.3% of 2,004 paired episodes the two interfaces led the agent down different paths, and early differences carried into later steps.
A fix that keeps the decision and repairs the call
The team also proposed DART. When a tool call fails, DART keeps the action the model already chose and only rewrites how that action is expressed for the interface. It does not ask the model to pick again.
On Gemini-2.5-Flash, DART raised scores in all six settings, by up to 9.0 points. Under MCP it also cut tokens per task by 27.1k on ALFWorld, 11.2k on WebShop and 4.3k on ScienceWorld. Under CLI the token change was small and went both ways.
What to keep in mind
The environments are research benchmarks, and the study covers small and mid-size open models plus one Gemini model. Nothing here tests Claude or GPT models, or real production tools. The DART numbers come from a single model.
The practical point is narrower than "MCP is bad" or "CLI is better". If you build an agent, test your tools in both forms with your own model before you commit. The code is in the authors' MICA repository.
Frequently asked questions
Is MCP better than CLI for AI agents?
The study finds no overall winner. The better interface depended on the model and the task, and sometimes flipped between tasks for the same model.
What is MCP in AI agents?
MCP, the Model Context Protocol, is a standard way for an agent to call tools through structured requests rather than typed commands.
Is the code for the MCP vs CLI study available?
Yes. The authors list a public repository called MICA on GitHub.
