Edition

RSS

ResearchAnalysis

Test-time compute changed the scaling roadmap. Here is what it costs

For a decade, progress meant bigger training runs. Now labs can trade inference compute for capability instead. That shifts the economics from capex at the lab to opex at the user, and it changes what "a better model" means.

By

3 min read

Illustration of a compute chart with one tall bar labelled train once beside a rising series of small bars labelled think per query, and an invoice card showing easy, medium and hard queries costing more each.
Illustration of a compute chart with one tall bar labelled train once beside a rising series of small bars labelled think per query, and an invoice card showing easy, medium and hard queries costing more each.

From 2020 to 2024 the industry ran on a single story: bigger training runs produce better models, predictably, along a power law. Kaplan’s scaling laws and the Chinchilla revision from Hoffmann and colleagues turned that story into a planning tool. You could estimate how much better the next model would be from how much compute you were prepared to buy.

The reasoning models that arrived from late 2024 introduced a second axis. If a model is trained to think longer, then for a fixed set of weights you can buy more accuracy at inference time, per query, by letting it generate more tokens. Snell and colleagues showed that on many tasks this is a better use of a marginal dollar than more pre-training. The roadmap did not break; it acquired a dimension. But the economics changed in ways that are still working through the industry.

From capex to opex

Under pure training scaling, the cost of intelligence was paid once, up front, by the lab. Serving the model was cheap and roughly constant per token. The customer paid for volume, not for quality.

Under test-time scaling, quality is bought per query. A hard question costs more to answer well than an easy one, and the customer sees that in their bill. Three consequences follow:

  1. Model tiers are becoming effort tiers. Vendors expose thinking budgets, “effort” parameters or fast/slow variants of the same weights. Choosing a model now includes choosing how much to spend per call.
  2. Margins depend on routing. Sending every query to the maximum effort setting is ruinous. The teams doing well route by difficulty, and difficulty estimation has become an engineering discipline of its own.
  3. The lab’s capital expenditure is no longer the only thing to track. Inference fleet size and utilisation matter as much as training clusters, and the labs are building both.

Two lanes: training a bigger model is paid once by the lab and is cheap and flat per token; letting the model think longer is paid per query by the customer and hard questions cost more

Figure 1: Both axes buy capability. Only one of them shows up on the customer’s invoice.

Benchmarks without a cost axis are misleading

A single accuracy number for a reasoning model is close to meaningless without the compute that produced it. The same model can score anywhere on a wide range depending on the budget, and the budget used for the marketing number is usually the maximum.

Responsible reporting now shows accuracy as a curve against tokens or dollars per problem. When comparing models, compare the curves, or at least compare points at matched cost. Our guide to reading a model card covers what to look for, and our editorial on benchmarks as a marketing channel covers why you will often not find it.

Pre-training did not stop

It is fashionable to say that pre-training scaling has “hit a wall”. The evidence is weaker than the slogan. What has happened is that the cheapest capability gains moved: for a period, reinforcement learning on reasoning and better post-training delivered more per dollar than the next order of magnitude of pre-training. Labs with the money continue to do both, and the largest training runs in history are underway. The models that lead over the next two years are likely to be the ones that combine a bigger base with more effective test-time scaling, not those that pick one.

What this means if you build on these models

  • Measure accuracy per dollar, not accuracy. Run your evaluation set at several effort levels and plot it. The right operating point is rarely the maximum.
  • Route. A cheap classifier that decides which queries need deep reasoning pays for itself quickly.
  • Cache and distil. Expensive reasoning on a recurring question should happen once. Use the reasoning model to produce answers or training data, then serve a fast model.
  • Budget for variance. Per-query cost now depends on the question. Set limits.

Decision diagram for an incoming query: a checkable answer means reasoning helps, a one-step lookup means a fast low-effort model, and a latency-sensitive interaction means reasoning offline and distilling into a fast model

Figure 2: Routing by query type is where most of the savings from test-time compute are captured.

The scaling story is still true. It is just that “scale” now has two meanings, and one of them shows up on your invoice.

Sources

  1. Kaplan et al., "Scaling Laws for Neural Language Models" arXiv
  2. Hoffmann et al., "Training Compute-Optimal Large Language Models" arXiv
  3. Snell et al., "Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters" arXiv
All research →

ResearchExplainer

What "reasoning" models actually do differently

Reasoning models are trained to spend tokens thinking before they answer. Here is what that training involves, why it works on some problems and not others, and how to decide when to pay for it.

4 min read