Test-time compute changed the scaling roadmap. Here is what it costs, and who pays

For a decade, buying a better model meant buying a bigger training run. Since late 2024 there has been a second way: keep the weights fixed and let the model spend more tokens on each query. On tasks with a checkable answer, that second axis buys accuracy at least as efficiently as more pre-training, and the labs now scale both at once.

The change matters because of who pays. Training is capital spent once, by the lab, and amortised into a per-token price that used to be flat. Thinking is spent per query, by the customer, and it scales with how hard the question is. A “smarter” answer is now often a more expensive answer, and the cheapest way to lose money with a reasoning model is to let every request think as hard as the hardest one.

This piece sets out what the second axis is, what the primary evidence says it buys, what it costs on the price lists of September 2026, where it does nothing, and how a team building on these models should decide when to pay for it.

The old roadmap: one axis, paid up front

Kaplan and colleagues at OpenAI wrote the planning document for the first era in 2020. Loss, they found, “scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude”. Larger models were more sample-efficient, so the compute-optimal move was to train very large models on relatively modest data and stop well before convergence.

DeepMind’s Chinchilla paper corrected the recipe in 2022 without changing the shape of the argument. Hoffmann and colleagues found that “for compute-optimal training, the model size and the number of training tokens should be scaled equally”, and demonstrated it with a 70-billion-parameter model trained on four times the data of the 280-billion-parameter Gopher, at the same compute budget. Chinchilla beat Gopher across the board, reaching 67.5% on MMLU, and because it was smaller it was cheaper to fine-tune and to serve.

That is the key economic fact of the first era. Under training-scaling laws, quality was decided at training time and the serving cost per token was roughly constant. The customer paid for volume. The lab paid for intelligence.

The money followed the law. Epoch AI’s trend data puts the growth of training compute for frontier language models at about 5x a year since 2020, doubling every 5.2 months, with the cost of frontier training runs growing about 3.5x a year over the same period. Epoch also notes that only around 30% of OpenAI’s compute spending in 2024 went on inference. The capital went into clusters.

The slowdown, when it came, was about data rather than compute. Ilya Sutskever, in his NeurIPS talk in December 2024, put it bluntly: “Pre-training as we know it will unquestionably end”, because “we’ve achieved peak data, and there will be no more”. He named three things that would come next: agents, synthetic data and “inference time compute, or maybe what’s been most recently, most vividly seen in o1”. Whether or not one accepts the strong version of that claim, the labs behaved as if the cheapest gains had moved.

The second axis: what test-time compute actually is

Test-time compute is any computation spent per query rather than per training run. For reasoning models it takes three forms, which are often combined.

Longer chains of thought. The model generates intermediate reasoning tokens before its answer. More tokens, on the right tasks, mean more chances to check, backtrack and try another approach. This is the mechanism reasoning models are trained to use, and our explainer on what reasoning models actually do covers it in detail.

Repeated sampling with verification. Sample many candidate answers and pick one, using an automatic checker where one exists (unit tests, a proof assistant, a numeric answer key), or a vote or reward model where one does not.

Search. Instead of sampling whole answers, sample partial steps, score them with a process reward model and expand the promising ones: best-of-N over steps, beam search, or lookahead.

Two 2024 papers measured what these buy, under stated conditions, before any frontier lab productised them.

Snell et al.: a smaller model can beat one 14x larger, on the right questions

Snell and colleagues at Berkeley and DeepMind fine-tuned PaLM 2-S* into a revision model (which iteratively improves its own answer) and a process reward model (which scores partial solutions), and studied both on the MATH benchmark. Their central move was to allocate test-time compute “compute-optimally”: choose the method and the budget per question based on an estimate of that question’s difficulty.

The findings that matter for planning are conditional. On easy questions (MATH levels 1 and 2) sequential revision worked best and beam search actually “degrades performance”; on hard questions (levels 3 and 4) beam search “consistently outperforms best-of-N”. Allocating adaptively, the compute-optimal strategy beat a best-of-N baseline “using up to 4x less test-time compute (e.g. 64 samples versus 256)”.

The headline comparison was FLOPs-matched: give a small model extra inference compute, or spend the same FLOPs pre-training a model 14x larger, and compare. The answer depended on two things. The first was difficulty: on easy and intermediate questions the small model with test-time compute won; on the hardest, “pretraining is a more effective way to improve performance”. The second was the ratio R of inference tokens to pre-training tokens, which the paper set at 0.16, 0.79 and 22. When a model will be used little relative to its training (low R), test-time compute wins. When it will serve enormous volumes (high R), the extra pre-training pays for itself. The authors’ own conclusion is that the two kinds of compute are “not 1-to-1 exchangeable”.

Brown et al.: coverage keeps climbing for four orders of magnitude

The Large Language Monkeys paper from Stanford and collaborators asked a simpler question: if you just sample more, how often does at least one sample solve the problem? On SWE-bench Lite, the fraction of issues solved by DeepSeek-Coder-V2-Instruct rose “from 15.9% with one sample to 56% with 250 samples”, beating the single-attempt state of the art of 43%. On MATH, Llama-3-8B-Instruct’s coverage climbed “from 79.8% with 100 samples to 95.3% with 10,000 samples”. The relationship between coverage and samples was “often log-linear” over four orders of magnitude.

The catch is in the selection step. Where an automatic verifier exists, such as unit tests for code, all of that coverage is usable. Where it does not, the paper found that “majority voting and reward model scoring plateau beyond several hundred samples” while coverage keeps rising. The model can produce the right answer; you cannot reliably find it. That gap is the single most important limit on the whole approach, and it recurs below.

Epoch AI had estimated the exchange rate in general terms in July 2023: for most language tasks, one to two orders of magnitude more inference compute could substitute for about one order of magnitude of training compute, and for tasks with cheap verification and parallel solving, such as coding and formal maths, the substitution could reach five or six orders of magnitude of inference for three or four of training. The 2024 papers put numbers and conditions on that estimate.

Figure 1: Both axes buy capability. Only one of them shows up on the customer’s invoice.

The frontier models turned the axis into a product

o1 and o3: the log-scale plots

OpenAI’s o1 announcement in September 2024 is the document that made the second axis official. The company wrote that “the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)”, and published two log-scale plots of AIME accuracy against each. The numbers were the point: on AIME 2024, GPT-4o solved 12% of problems; o1 solved 74% with a single sample, 83% with consensus among 64 samples, and 93% “when re-ranking 1000 samples with a learned scoring function”. Note that the last two figures are exactly the repeated-sampling-plus-selection recipe from Brown et al., and that the 93% is the number that travelled.

In April 2025 the o3 and o4-mini post described the recipe as a deliberate scaling programme: “By retracing the scaling path, this time in RL, we’ve pushed an additional order of magnitude in both training compute and inference-time reasoning, yet still see clear performance gains.” It added that at equal latency and cost with o1, o3 scored higher, “and we’ve validated that if we let it think longer, its performance keeps climbing”.

The ARC Prize bill

The most useful public record of what that climb costs came from a third party. In December 2024 the ARC Prize Foundation tested a preview of o3 on its semi-private ARC-AGI-1 set under two configurations. At high efficiency, six samples per task and 33.5 million tokens for the 100 tasks, o3 scored 75.7%. At low efficiency, 1,024 samples per task and 5.7 billion tokens, it scored 87.5%, which ARC described as roughly 172x the compute of the cheaper run. The dollar figures on ARC’s page have been revised as OpenAI’s pricing for that class of model changed, most recently to o3-pro rates in December 2025; as currently published they are $26 per task for the high-efficiency run and $4,560 per task for the low-efficiency one. The foundation stated that only the high-efficiency score qualified for its public leaderboard, whose rules cap runs at $10,000. Whichever revision one uses, the shape is the same: twelve points of accuracy for two orders of magnitude of cost.

ARC also predicted that its harder ARC-AGI-2 set would drop o3 “to under 30%”. At the ARC-AGI-2 launch in March 2025 the in-progress estimate for o3-preview at low effort was 4%, while a panel of humans solved every task at about $17 per task.

DeepSeek-R1: the recipe in the open

DeepSeek’s R1 paper in January 2025 showed how the behaviour is trained, not just how it is priced. DeepSeek-R1-Zero was trained from DeepSeek-V3-Base with reinforcement learning alone, no supervised fine-tuning, using GRPO and rule-based rewards for accuracy (a checkable answer, or code that compiles and passes) and for format. The paper’s Figure 3 shows the average response length rising steadily through training: the model learned to think longer because longer thinking was rewarded. Its AIME 2024 pass@1 rose “from an initial 15.6% to an impressive 71.0%”, and to 86.7% with majority voting.

The released DeepSeek-R1 added a four-stage pipeline (a small cold-start SFT set, reasoning RL, rejection-sampled SFT on about 800,000 examples, and a final RL pass) and reported 79.8% on AIME 2024, 97.3% on MATH-500 and a 2,029 Codeforces rating. Two details matter for the economics. First, the reward signal was a verifier: the recipe works where answers can be checked. Second, the paper’s Table 6 shows that a 32-billion-parameter Qwen model distilled from R1 scored 72.6% on AIME, while the same base trained with RL from scratch reached 47.0%. Distillation captured the behaviour more cheaply than reproducing it, a point we return to at the end.

s1: budget forcing with a thousand examples

The s1 paper from Stanford and collaborators, also January 2025, showed how little is needed to expose the dial. The authors fine-tuned Qwen2.5-32B-Instruct on 1,000 curated questions with reasoning traces, then controlled thinking length at inference with “budget forcing”: cutting the model off to force an answer, or appending the token “Wait” to make it keep going. The resulting s1-32B exceeded o1-preview “by up to 27%” on MATH and AIME24, and budget forcing let it extrapolate past its unforced performance, “from 50% to 57% on AIME24”. The dial is real, it is cheap to install, and every vendor now ships one.

The economics: capex at the lab becomes opex at the customer

How thinking is billed

The billing rule is the same everywhere as of September 2026: reasoning tokens are output tokens. Anthropic’s documentation lists “tokens Claude uses while thinking (billed as output tokens)” and adds that thinking blocks from earlier turns that remain in context are billed as input. OpenAI’s reasoning guide says reasoning tokens “are billed as output tokens” and “occupy space in the model’s context window” even though they “are not visible via the API”, and recommends “reserving at least 25,000 tokens for reasoning and outputs” when starting out. Google’s pricing page labels every output price “including thinking tokens”, and its thinking documentation warns that “pricing is based on the full thought tokens the model needs to generate, despite only the summary being output from the API”.

The controls have converged too. Anthropic’s manual budget_tokens (minimum 1,024, a target rather than a cap, with a recommendation to use batch processing above 32,000 because longer requests “can hit system timeouts”) is deprecated on its 4.6 models and rejected from 4.7 onwards in favour of adaptive thinking steered by an effort parameter with five levels: low, medium, high, xhigh and max. OpenAI’s reasoning.effort runs from none through minimal, low, medium, high and xhigh to max, depending on the model, with GPT-5.5 defaulting to medium. Google exposes thinking_level at low, medium and high. One set of weights, several price points, chosen per request.

Prices from each vendor’s pricing page, read in September 2026; list rates, before caching, batch or regional multipliers. Anthropic notes that its 4.7-and-later tokenizer produces roughly 30% more tokens for the same text than earlier models, which is a price rise that does not show in the table.

One task at three thinking budgets

Take a real but ordinary job: a 3,000-token prompt (a support ticket, the relevant policy text and instructions) that needs an 800-token answer, run on Claude Opus 5.5 at $4 per million input and $20 per million output tokens. Assume the thinking spend at three effort settings is 2,000, 8,000 and 24,000 tokens. Those three figures are illustrative assumptions, not measurements; the shape of the arithmetic is what matters.

• Low. Input $0.012, output 2,800 tokens at $0.056. About $0.07 per task.
• Medium. Input $0.012, output 8,800 tokens at $0.176. About $0.19 per task.
• High. Input $0.012, output 24,800 tokens at $0.496. About $0.51 per task.

The visible answer is identical in length in all three cases. The bill differs by 7x, and almost all of the difference is tokens nobody reads. At 1,000 tasks a day that is roughly $2,000, $5,600 and $15,000 a month for what may or may not be a different answer. The only way to know is to measure accuracy at each setting on your own data, which is the routing question below. Our piece on the real cost of an AI product walks through how this line item interacts with the rest of the budget.

The latency bill

Thinking tokens are generated sequentially, so the second cost is time. Anthropic’s documentation says plainly that higher budgets come “at the cost of increased latency”, and its own measurements make the scale concrete: on its DeepWideSearch benchmark, Claude Fable 5 at low effort took 4.5 minutes per problem against 7.9 minutes at the default, and on a long-horizon corpus benchmark Fable 5.1 episodes took 15.2 hours at low effort and 19.9 hours at high. Epoch AI’s inference-economics analysis explains why serving faster is not free either: absent latency constraints a provider could “quadruple inference speed for each doubling of cost per token” by spreading a forward pass across more GPUs, but in practice fixed per-token latency limits how far that goes. For an interactive product, a thinking budget is a waiting-time budget as much as a dollar budget.

What it changed in the roadmap

Labs scale both axes

The labs’ stated plans are explicit about running two programmes at once. OpenAI’s o3 post described “an additional order of magnitude in both training compute and inference-time reasoning”. Anthropic’s Opus 5 announcement in July 2026 reports that the model reaches its trading-benchmark result “using roughly a seventh of the reasoning tokens and under half the latency of Opus 4.8”, and that on CursorBench 3.2 at max effort it “performs within 0.5% of Fable 5’s peak score, but at half the cost per task”. OpenAI’s GPT-5.6 post makes the same kind of claim in the other direction, that Sol “with max reasoning comes within one point of Fable 5 while completing tasks in 61% less time at roughly half the estimated cost”, and reports Terminal-Bench 2.1 at 88.8% for Sol and 91.9% for a Sol “Ultra” mode that runs four agents in parallel. These are vendor claims, not independent measurements, but the framing is the tell: every launch is now reported as a point on a cost-versus-accuracy curve, and the cheapest lever for moving along that curve is effort.

Hardware demand moved to inference

Nvidia’s fourth-quarter fiscal 2025 release in February 2025 put the industry’s view in one sentence from Jensen Huang, who said reasoning AI “adds another scaling law”, in which “increasing compute for training makes models smarter and increasing compute for long thinking makes the answer smarter”. Data-centre revenue that quarter was $35.6 billion. By the second quarter of fiscal 2027, reported in August 2026, it was $89.0 billion, up 117% year on year, and Huang’s framing had shifted to the output: “Its tokens are productive and profitable.” Epoch’s figure that inference was about 30% of OpenAI’s 2024 compute spend is the baseline to watch; a world in which every query can think is a world in which that share rises.

Pricing tiers became effort tiers

In the first era a vendor’s line-up was small, medium and large models at three prices. In this one, each model is itself a range. Anthropic’s effort page describes max as “absolute maximum capability with no constraints on token spending”, xhigh as suited to “long-running agentic and coding tasks (over 30 minutes) with token budgets in the millions”, and low as “significant token savings with some capability reduction”. OpenAI sells a separate Pro tier at six times the list price that “performs more model work than standard mode”. A model choice is now a two-dimensional choice, and the effort dimension is usually the one with the bigger cost range.

Where it does not help

Tasks without a checkable answer

Every result in the evidence table above came from a domain with a verifier: competition maths with a numeric key, code with tests, ARC grids with a known output. That is not a coincidence. RL for reasoning needs a reward, repeated sampling needs a selector, and search needs a scorer. Brown et al.’s finding that voting and reward models “plateau beyond several hundred samples” without an automatic verifier is the ceiling for everything else: summarisation, drafting, judgement calls, open-ended advice. On those tasks a longer chain of thought may still help a little, but there is no mechanism by which spending 100x more can reliably buy 12 more points.

Diminishing returns and overthinking

Even with a verifier, the curve bends. Snell et al. found beam search hurt on easy questions. Two later papers measured the harm directly. Chen and colleagues’ overthinking study found that o1-style models spent “1,953% more tokens” than conventional models on the problem “2+3=?”, with one model generating 13 separate solutions, and that “in more than 92% of cases, the initial round of solutions produces the correct answer”. Their mitigation cut token use by 48.6% on MATH500 with no accuracy loss. Hassid and colleagues went further: across the models they tested, the shortest of several sampled reasoning chains was “up to 34.5% more accurate than the longest chain sampled”, and a method that stops early and votes among the first chains to finish used “up to 40% fewer thinking tokens” while matching or beating standard majority voting.

The vendors’ own curves say the same

The most persuasive evidence that returns diminish comes from the sellers. Anthropic’s cost-optimisation guide reports that on four research and knowledge-work benchmarks, Claude Fable 5 at low effort “gave up 1 to 3 points for one-third to one-half off the cost per task”, medium matched the default “at about 70% to 87% of its cost”, and the default high “bought nothing measurable over medium on any of the four”. On DeepResearch Bench II, Claude Fable 5.1 scored nearly the same at low, medium and high while cost per task rose from $4.66 to $7.12. On SWE-bench Pro, by contrast, Claude Opus 5.5 at medium was about 2.5 points below high at roughly 70% of the cost, low was about 8 points below at a third of the cost, and xhigh was about 1.4 points above at 2.5x the cost. The same document states the principle: “below the model’s ceiling, the highest effort levels pay for depth the task never uses”. Anthropic’s guidance for Opus 4.7 says max “adds significant cost for relatively small quality gains” and “can lead to overthinking” on structured-output tasks.

The ARC Prize leaderboard shows it in public. As of September 2026, Claude Opus 5.5 at high effort scores 93.3% on ARC-AGI-2 at $0.408 per task; the same model at xhigh scores 92.5% at $0.671 per task. More thinking, lower score, 64% higher bill. This is why a single accuracy number is close to meaningless for a reasoning model, and why our editorial on benchmarks as a marketing channel argues for curves, or at least matched-cost points, over headline scores.

Figure 2: The effort dial only earns its cost in the first column. The other two are where most production traffic lives.

The routing decision for builders

The savings from test-time compute are captured by not spending it. Three questions decide whether a query deserves thinking, and they can be asked cheaply, by a small classifier, a rule or the caller’s own knowledge of the workload.

Four practices follow from the evidence.

Measure the curve, then pick a point. Run your evaluation set at three effort levels and record accuracy, tokens and wall time. Anthropic’s own advice is to “run an effort sweep on your own evals rather than carrying settings over from an earlier model”, because the same setting thinks more on a newer model. The right operating point is rarely the maximum and is often the middle.

Route before you think. A cheap difficulty estimate up front, which is exactly what Snell et al. used to allocate compute, turns a flat 7x bill into a 7x bill on the tenth of traffic that needs it.

Cache and distil recurring work. Expensive reasoning on a recurring question should happen once. DeepSeek’s result that a distilled 32B model scored 72.6% on AIME where RL from scratch reached 47.0% is the template: pay for thinking to produce labels or traces, then serve something fast.

Budget for variance. Per-query cost now depends on the question, and effort is, in Anthropic’s words, “a behavioral signal, not a strict token budget”. Set max_tokens as the hard cap, watch the thinking-token field in usage reports, and alert on the tail.

Figure 3: Routing by query type is where most of the savings from test-time compute are captured.

The counter-argument: a stopgap the base model absorbs

The strongest case against building a roadmap around test-time compute is that its price is falling faster than anything else in the industry. Epoch AI’s “plunging price of thought” analysis, published in September 2026, finds that the cost of a given level of performance on reasoning benchmarks has fallen about 47% per quarter, or 13x a year, since March 2023. An earlier Epoch analysis of inference prices found declines of 9x to 900x a year depending on the benchmark. Epoch’s February 2026 note on the inference cost burden gives a concrete instance: reaching about 27% on FrontierMath took roughly 43 million output tokens with o4-mini in April 2025 and about 5 million with GPT-5.2 eight months later. Our news analysis on why inference prices keep falling covers the hardware and competition behind those numbers.

The ARC Prize record makes the same argument in one benchmark. In December 2024, 87.5% on ARC-AGI-1 cost about $4,560 per task. In September 2026 the leaderboard shows Claude Opus 5.5 at high effort scoring 98.5% on the same benchmark at $0.158 per task. What was bought with a thousand samples is now bought with a better model at a fraction of a cent of the old price. On that reading, test-time compute is the way a lab demonstrates a capability a year before it can train it in, and distillation (R1 into Qwen, o1’s 93% into what any mid-tier model does today) is the way the capability moves down the price list.

We think the argument is right about any single capability and wrong about the axis. Every specific thing you pay for with thinking tokens today will be cheap by next year. But the frontier moves too, and the tasks worth paying for move with it: long-horizon agent runs, hard code changes, open research questions. Snell et al.’s own condition still holds: pre-training wins on the hardest questions and at the highest volumes, which is why the labs have not stopped training bigger models, and test-time compute wins on the questions just past what the base model can do, which is where every new product tries to live. The dial does not go away. It gets cheaper, and the things you point it at get harder.

What to take from this

The scaling story is still true, and it now has two axes. One is paid once, by the lab, and shows up as a better default model every few months. The other is paid per query, by whoever calls the API, and shows up as a line in the usage report that can be zero or can be most of the bill.

For anyone building on these models, the practical consequences are three. Report accuracy against cost, never accuracy alone, because a single number hides the budget that produced it. Route by task and re-measure each quarter, because the point where thinking stops paying moves with every release. And treat expensive thinking as a way to generate something cheaper, a label, a trace, a cached answer, rather than as a permanent operating cost. The roadmap did not break in 2024. It acquired a dimension, and the dimension is on your invoice.