# How reasoning models work: more thinking time, trained by reward > Chain of thought, extra compute at answer time and reinforcement learning explain most of it. Here are the numbers from the papers. URL: https://claudegpt.si/news/how-do-reasoning-models-work Published: 2026-11-01T06:51:22.913Z Author: ClaudeGPT Si Newsroom Section: Research Topics: DeepSeek, Anthropic, Reasoning models, Reinforcement learning, Chain of thought Publisher: ClaudeGPT Si ## Key takeaways - Reasoning models write a long chain of working before answering, and are trained with rewards to make that working useful. - DeepSeek reports R1-Zero's AIME 2024 first-try pass rate rose from 15.6% to 77.9% during reinforcement learning, with no human-written reasoning examples. - Snell and colleagues found a smaller model given extra thinking time beat a model 14 times larger at matched compute on problems of suitable difficulty. A reasoning model is a language model that writes out a long chain of working before it gives its answer, and is trained to make that working useful. That is the short answer to how reasoning models work. Three published ideas explain most of it: [chain-of-thought prompting](https://arxiv.org/abs/2201.11903), spending more computing power at answer time, and [reinforcement learning on problems with checkable answers](https://arxiv.org/abs/2501.12948). ## Step one: show your work In January 2022, Jason Wei and colleagues showed that a large model does better on math and logic problems if it writes intermediate steps first. A step-by-step solution is called a chain of thought. They put eight worked examples in the prompt, and a 540-billion-parameter model reached the best known accuracy on GSM8K, a set of grade-school math word problems. It beat a fine-tuned GPT-3 that had a separate checker model. The paper also found that this ability shows up only in sufficiently large models. ## Step two: spend more computing power while answering Writing more tokens costs more computing power for each question. This is called test-time compute, meaning the work done while answering rather than while training. In August 2024, Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar studied how to spend that budget. They tried two methods: searching over many candidate solutions with a verifier that scores each step, and letting the model revise its own answer. The best choice depended on how hard the question was. Their "compute-optimal" strategy, which gives hard questions more effort and easy ones less, used more than four times less compute than simply sampling N answers and picking the best. They also reported a result that explains the industry's interest. On problems where a smaller model already had a reasonable success rate, extra thinking time let it beat a model 14 times larger when both used the same total compute. The authors compared these under matched compute, so it is not a claim that thinking always beats size. ## Step three: train the model to reason with rewards Prompting a model to show its work is one thing. Training it to do so is what makes a reasoning model. The clearest public account is the [DeepSeek-R1 paper](https://arxiv.org/abs/2501.12948), first posted January 22, 2025 and later published in Nature. The team used reinforcement learning, which means the model tries things and gets a score. The score came from simple rules. One reward checks whether the final answer is correct, using a rule for math or a compiler for code. A second checks that the reasoning sits inside designated think tags. The two are weighted equally. No human wrote out reasoning for the model to copy in this first version, called DeepSeek-R1-Zero. The training method, Group Relative Policy Optimization, samples 16 answers per question and rewards the ones that did better than their group. DeepSeek reports that on AIME 2024, a hard high-school math competition, R1-Zero's pass rate on the first try rose from 15.6% to 77.9% during training. With majority voting over many samples, it reached 86.7%. The model also changed how it wrote. Its answers grew to hundreds or thousands of tokens as training went on. The authors describe an "aha moment" in which the model began stopping mid-solution and rethinking, with a sudden rise in the word "wait". Nobody programmed that. It appeared because backtracking led to more correct answers, and correct answers earned reward. The released DeepSeek-R1 adds steps around this: a cold start on a few thousand human-aligned long examples, a reward for staying in one language, and a second round of training for helpfulness and safety. ## What you see as a user Commercial products wrap the same idea in settings. Anthropic's [extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) lets a developer set a thinking budget in tokens. The minimum is 1,024. The budget is a target, not a hard cap, and the model may stop sooner. Thinking tokens count toward the output limit and show up in the billed output tokens. The docs say that extended thinking is now a legacy feature, and that newer Claude models use adaptive thinking, where the model decides whether and how much to think, guided by an effort setting. This is as of November 1, 2026. ## Limits The DeepSeek authors list several. The model overthinks simple questions. Few-shot prompting makes it worse. It mixes languages outside Chinese and English. Tool use and structured output trail other models. Reinforcement learning also needs answers that can be checked automatically, so it works best on math, code and science. For tasks like writing, the authors say building a reliable reward is hard and models can learn to game it. ## Where to look next The practical test for any reasoning model is cost against accuracy on your own tasks: compare the same prompt with thinking off, a small budget and a large one, and watch the thinking-token count in the usage fields. ## Frequently asked questions ### What is test-time compute? It is the computing power a model uses while answering a question, as opposed to while it is being trained. Reasoning models spend more of it by writing longer chains of thought or trying several answers. ### Do reasoning models always think longer? Not by choice of the user. Anthropic's docs say thinking budgets are a target, and newer Claude models use adaptive thinking, deciding per request whether and how much to think. DeepSeek's authors note that their model still overthinks simple questions. ### Why do reasoning models cost more? The thinking is made of tokens. Anthropic's docs say thinking tokens count toward the output limit and appear in billed output tokens, so longer thinking means a larger bill. ### Are reasoning models better at everything? No. Reinforcement learning works best where answers can be checked automatically, such as math and code. DeepSeek's authors say reliable rewards are hard to build for tasks like writing. ## Sources - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948), arXiv / DeepSeek-AI - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903), arXiv - [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://arxiv.org/abs/2408.03314), arXiv - [Extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking), Anthropic