# TAC-CoT lets small models reason in hidden tokens and call tools

> A new EMNLP 2026 paper reports up to 8.8 points higher accuracy and 6.8 times fewer forward passes than explicit chain-of-thought, on models of 0.5 to 4 billion parameters.

URL: https://claudegpt.si/news/tac-cot-tool-empowered-implicit-reasoning  
Published: 2026-11-05T14:52:37.691Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: TAC-CoT, Chain of thought, Tool use, Hugging Face  
Publisher: ClaudeGPT Si

## Key takeaways

- TAC-CoT keeps tool calls, results and answers as text but replaces written reasoning with short latent segments.
- On Qwen2.5-0.5B, Llama-3.2-1B and Qwen3-4B it improves math accuracy by 5.8 points and QA exact match by 2.4 points on average over explicit chain-of-thought.
- The arXiv page lists no code, and only models up to 4 billion parameters were tested.

TAC-CoT is a method from Juseon Do, Sangmin Bae, Jeesu Jung and Hwanjun Song that lets small language models reason in short hidden steps while still calling tools such as a calculator or a search engine. In [the paper, posted to arXiv on November 1](https://arxiv.org/abs/2611.01926), the authors report up to 8.8 points higher accuracy than ordinary chain-of-thought and up to 6.8 times fewer forward passes. It is accepted to EMNLP 2026 as a main-conference long paper.

The paper has 10 upvotes on Hugging Face daily papers. The numbers come from the authors' own experiments on models of 0.5 to 4 billion parameters.

## The problem: thinking in text is slow, hiding it breaks tools

Chain-of-thought (CoT) means a model writes out its reasoning before it answers. It helps on math and multi-step questions, but every written word costs one forward pass through the network, and long traces can pile up errors.

Implicit CoT tries to fix the cost. Instead of words, the model produces a few continuous vectors, called latent tokens, that carry the reasoning. That is cheaper. But earlier implicit methods, such as CODI, PCCoT and SIM-CoT, put one fixed block of latent tokens before the answer. A model that works this way can't stop halfway, call a tool, read the result and keep thinking. It also spends the same budget on an easy question and a hard one.

## What TAC-CoT does

TAC-CoT interleaves the two modes. Each reasoning segment is a short run of latent tokens (3 by default, 2 for the largest model). Tool calls, tool results and the final answer stay as plain text.

After each segment, the model compares two scores: how likely a tool call is, and how likely an answer is. If the tool call wins, it calls the tool, appends the result and starts another latent segment. Otherwise it answers. The number of segments is set at inference time, so hard questions can get more.

Training combines three pieces. A student model learns to reconstruct the hidden reasoning with an auxiliary decoder. A teacher model trains on full explicit traces. A distillation loss pulls the student's internal state toward the teacher's at the point where the answer begins.

The math data is GSM8K-Aug with calculator equations. For question answering, the team built 40,000 HotpotQA trajectories with DeepSeek-V3 and Wikipedia retrieval, keeping only those with the correct answer. They report that generation cost $102.73 and that a GPT-5.1 audit of 200 samples judged 99% valid.

## Results

The paper tests Qwen2.5-0.5B-Instruct, Llama-3.2-1B-Instruct and Qwen3-4B on GSM8K, GSM-Hard and SVAMP for math, and HotpotQA, 2WikiMultihopQA and KILT-TriviaQA for question answering, using 1,000 random examples per dataset. The baseline is explicit CoT with the same tools.

| Backbone | Math accuracy | QA exact match | Forward passes, math | Forward passes, QA |
| --- | --- | --- | --- | --- |
| Qwen2.5-0.5B | 48.4 to 53.9 | 42.4 to 45.8 | 104.4 to 42.4 | 283.3 to 44.2 |
| Llama-3.2-1B | 53.4 to 59.2 | 51.1 to 53.1 | 76.3 to 35.1 | 245.3 to 48.0 |
| Qwen3-4B | 68.5 to 74.5 | 58.5 to 60.6 | 110.4 to 40.4 | 244.0 to 41.8 |

Each cell reads explicit CoT to TAC-CoT. Averaged over the three models, math improves by 5.8 points and QA by 2.4. The gains hold out of domain: 5.5 points on math sets the model wasn't trained on, and 2.5 on QA.

Other findings from the paper:

- **Against the fixed-budget baselines.** TokenSkip, PCCoT, CODI and SIM-CoT use few forward passes, but on QA they fall far behind because they can't retrieve evidence. For Qwen2.5-0.5B without tools, their QA scores sit around 5 to 15.
- **Harder problems gain more.** Over SIM-CoT, the improvement grows from 4.6 points on one-step GSM8K problems to 30.3 points on four-step ones.
- **Tool use.** On ToolBench, TAC-CoT beats explicit CoT in 51.8% of solvable single-tool cases and 58.6% of multi-tool cases. The multi-tool result is significant at p<0.01.
- **Speed in practice.** On an L40S GPU, wall-clock speedup is 2.3 times on math and 2.4 times on QA. That is smaller than the forward-pass reduction, and the authors warn the two don't map directly.
- **Controls.** Compressing explicit CoT to half its length with TokenSkip and tools scored 45.4 on the Qwen2.5-0.5B math average, against 53.9 for TAC-CoT. Setting the latent tokens per segment to zero gave 42.0, against 45.4 with three, in the no-tools ablation.

## What to be careful about

The models are small. The largest is 4 billion parameters, and the paper doesn't show whether the gains hold for frontier-size models, which are already strong at math and retrieval.

Only two kinds of task are covered: grade-school math with a calculator, and Wikipedia question answering. ToolBench is reported as a win rate against explicit CoT, not an absolute score. The authors list two limits themselves: routing happens per segment rather than per token, and longer interactions haven't been tested for real deployment.

The explicit baseline is also trained on the same data, so the comparison says TAC-CoT beats that recipe, not that it beats any larger off-the-shelf model.

## Why it matters

Agents spend most of their cost generating text between tool calls. If short latent steps can replace most of that text without hurting tool use, small on-device or low-cost agents get cheaper to run. This paper is evidence that it can work at small scale. It is not yet evidence for production systems.


## Code and paper

The arXiv page lists no code repository, and the paper text states no code release. Check [the arXiv listing](https://arxiv.org/abs/2611.01926) for updates; the [HTML version](https://arxiv.org/html/2611.01926) has the full tables.

## Frequently asked questions

### What is implicit chain-of-thought?

It replaces written reasoning with a few continuous vectors that carry the reasoning inside the model. That costs fewer forward passes than writing out every step.

### What does TAC-CoT add to implicit chain-of-thought?

Earlier methods used one fixed block of latent tokens and could not call tools mid-reasoning. TAC-CoT alternates short latent segments with explicit tool calls and results, and chooses at each step whether to call a tool or answer.

### Is TAC-CoT code available?

The arXiv page lists no code repository, and the full text read for this article shows no release statement.

## Sources

- [TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens (abstract)](https://arxiv.org/abs/2611.01926), arXiv
- [TAC-CoT (full text)](https://arxiv.org/html/2611.01926), arXiv