What "reasoning" models actually do differently

Since late 2024 every major lab has shipped a “reasoning” model, and the term has taken on a slightly mystical tone in product copy. The mechanism is less mysterious than the marketing, and understanding it is the difference between paying for reasoning where it helps and paying for it everywhere.

The mechanism

A reasoning model is an ordinary large language model that has been post-trained to generate a long sequence of intermediate tokens, a chain of thought, before it commits to an answer. The core observation, documented by Wei and colleagues in 2022, is that prompting a model to “think step by step” improves accuracy on multi-step problems. Reasoning models bake that behaviour in and, crucially, train it.

The training recipe that made the current generation possible is reinforcement learning on verifiable tasks. You take problems with checkable answers, such as maths competition questions, unit-tested coding tasks and logic puzzles, let the model generate long attempts, and reward the attempts that reach the right answer. The model learns, without step-by-step supervision, to plan, to check its own work, to backtrack when a line of attack fails and to spend more tokens on harder problems. DeepSeek’s R1 report was notable for showing how much of this emerges from a fairly simple reward.

The result is a model that converts extra inference compute into extra accuracy. That is the property Snell and colleagues formalised as test-time compute scaling, and it is why “how long did it think” is now a dial you can turn.

Figure 1: Reinforcement learning on checkable problems teaches the model to spend more tokens when a problem is hard.

Where it helps

The gains concentrate on problems with two properties: the answer can be checked, and reaching it takes many dependent steps.

• Mathematics and formal logic. The canonical case; the improvements are dramatic.
• Code generation and debugging, especially when the model can run tests and iterate.
• Structured planning: decomposing a task, sequencing tool calls, satisfying constraints.
• Careful reading of long inputs where the answer depends on reconciling several passages.

Where it does not

• Recall. Knowing a fact does not benefit from thinking longer. Reasoning models are not more knowledgeable than their base models.
• Style, taste and tone. Writing tasks often get worse, or at least more laboured.
• Latency-sensitive interactions. A chatbot that pauses for twenty seconds is a worse chatbot.
• Simple classification and extraction at volume, where the extra tokens are pure cost.

Vendors increasingly ship a “thinking budget” or effort setting because of this. The same model with a low budget behaves like a fast conventional model; with a high budget it becomes a slow expert. Measure each production task at two or three budgets before choosing.

Figure 2: The dividing line is whether the answer can be checked and whether reaching it takes several dependent steps.

What the chain of thought is not

It is tempting to read the visible reasoning as a transcript of what the model “really did”. Interpretability research has repeatedly shown that chains of thought can be unfaithful: the model may reach an answer for reasons that do not appear in the text, and the text may contain steps that did not influence the outcome. Some vendors now show summarised or hidden reasoning for this reason among others.

The practical rule: use the chain of thought for debugging, prompt design and building intuition. Do not build product logic that parses it, and do not present it to users as an explanation of the answer.

Deciding when to pay

A rough decision procedure that has held up well in practice:

1. Does the task have a checkable answer, or at least a clear success criterion? If not, reasoning is unlikely to help much.
2. Does solving it require holding several intermediate results? If it is a one-step lookup, no.
3. Can the user tolerate the latency? If not, consider running the reasoning model offline to generate training data or few-shot examples for a fast model instead.
4. Have you measured the accuracy gain against the cost increase on your own data? If not, you are guessing.

Reasoning models are a genuine capability step, arguably the most important since instruction tuning. They are also the easiest way to quadruple an inference bill for no benefit. The mechanism tells you which one you are getting. For the roadmap implications, see how test-time compute changed the scaling picture.