Inference prices keep falling. Here is who actually keeps the saving

The price of a fixed amount of language-model capability has been falling faster than any input cost in the history of computing. Epoch AI, tracking the cheapest model that clears a given benchmark score, finds declines of between 9 and 900 times per year depending on the task. Stanford’s AI Index puts one concrete case on it: querying a model that scores as well as GPT-3.5 on MMLU cost about $20 per million tokens in November 2022 and $0.07 by October 2024, a 280-fold drop in 23 months.

That is the headline. The useful question for anyone running a product on these APIs is narrower: who keeps the money? A list-price cut becomes margin only when a product’s spend is token-bound, its volume is fixed and its quality bar is already met. For everyone else the cut becomes more steps, more context, more reasoning effort and a bill that stays roughly where it was.

This piece prices twelve current models as read on 27 September 2026, separates the headline decline from the discounts and surcharges that actually move a bill, works one product through three price points, and ends with what a team should do about it. We think the decline is real and will continue, and we think most teams are capturing far less of it than they could.

How fast prices have fallen, and what “same capability” hides

Three independent analyses agree on the direction and roughly on the slope.

Epoch AI’s March 2025 data insight, LLM inference prices have fallen rapidly but unequally across tasks, tracks the lowest-priced model that reaches a fixed score on six benchmarks (MMLU, GPQA Diamond, MATH-500, MATH level 5, HumanEval and Chatbot Arena Elo) over three years. Its headline is that the price to match GPT-4’s performance on GPQA Diamond, a set of PhD-level science questions, fell by 40 times per year, and that across all six milestones the rate of decline varies “from 9x to 900x per year”. The tables include open-weight models as priced by third-party hosts, not self-hosted costs.

Andreessen Horowitz’s LLMflation post from November 2024, by Guido Appenzeller, uses a simpler method: hold MMLU constant and read off the cheapest price. At an MMLU score of 42, GPT-3 cost $60 per million tokens in November 2021 and Llama 3.2 3B on Together cost $0.06 three years later. At the GPT-4 threshold of MMLU 83, prices had fallen about 62 times since March 2023. The post’s summary is “10x every year”, which it notes is faster than compute cost during the PC era or bandwidth during the dotcom boom. Its sharpest aside: OpenAI’s o1, then the flagship, cost the same per output token as GPT-3 at launch. As of 27 September 2026, o1 is still listed on OpenAI’s pricing page at $15 input and $60 output per million tokens, so the observation still holds.

Stanford’s AI Index 2025 repeats the Epoch range and adds the GPT-3.5 case quoted above.

The caveats that matter

All three measure the price of yesterday’s capability, and that is a moving target in two directions.

First, “the cheapest model that clears the bar” is a floor drawn by whichever vendor is most aggressive in that month, often an open-weight model on a discount host. It is not what most teams pay. A team that stays on the flagship tier sees a different curve, one that is closer to flat, as the price table below shows.

Second, the benchmark thresholds are static while the tasks people actually run are not. Nobody in 2026 is buying MMLU 42. The demand curve moves up as the supply curve moves down, which is why the two things can both be true: constant-capability prices fall 10 times a year, and the median AI product’s bill does not.

Epoch itself flags a third: “The fastest price drops in that range have occurred in the past year, so it’s less clear that those will persist.” We take that seriously in the counter-argument section.

Where the cuts land first: the September 2026 price table

The pattern in vendor line-ups has held for three generations now. The small tier is cut hardest and most often. The mid tier is replaced by a better model at the same or a slightly lower price. The flagship holds its price and gets superseded. And a new premium tier appears above it.

The table lists current list prices per million tokens for the standard (non-batch) tier, read on 27 September 2026 from each vendor’s own pricing page. Cache-read prices are the vendor’s price for a cached input token. Open-weight rows are priced by the host named.

Prices are per million tokens in USD, standard tier, text input. “n/a” means the vendor page does not list a separate cached-input price for that model, not that caching is unavailable. Gemini and xAI charge double on prompts above 200k tokens.

What the table shows

The small tier has been cut in half twice in a summer. OpenAI’s changelog records GPT-5.6 launching on 9 July 2026, then “Starting July 30, GPT-5.6 Luna costs 80% less, while GPT-5.6 Terra costs 20% less.” The current $0.20 and $1.20 for Luna imply a launch price of $1 and $6. GPT-6 Luna, released on 22 September 2026, lists at $0.10 and $0.50, so the entry model is a tenth of its price of eleven weeks earlier. Anthropic has not followed on the small tier: Claude Haiku 4.5 is still $1 and $5, ten times GPT-6 Luna on input.

The mid tier is where “same price, better model” happens. Google’s Gemini API changelog shows 3.6 Flash (21 July), 3.7 Flash (13 August) and 3.8 Flash (2 September) all launching at an introductory $0.75 input and $3.75 output through 31 December 2026, with a standard price of $1.50 and $7.50 thereafter. Gemini 3.5 Flash, released 19 May, is $1.50 and $9.00. Google’s own launch post for 3.6 Flash frames the gain as cost per task rather than per token: it “reduces the overall cost per agentic task” and, citing the Artificial Analysis index, uses “17% fewer output tokens than 3.5 Flash”. Anthropic’s Sonnet 5 launched at $2 and $10, described at launch as introductory pricing with a rise to $3 and $15 scheduled for 1 September; on 10 August the company made the lower price permanent. OpenAI’s GPT-6 Sol matches it exactly at $2 and $10, half of GPT-5.6 Sol’s $4 and $20.

The flagship holds and a new tier appears above it. Claude Fable 5.1 and Claude Fable 5 both list at $10 and $50. GPT-6 Astra, launched 3 September, lists at the same $10 and $50. Above those, OpenAI’s pricing page lists GPT-5.5 Pro and GPT-5.4 Pro at $30 and $180, and o1-pro at $150 and $600. Anthropic’s “fast mode” prices Opus 5.5 at $8 and $40, double the standard rate, and OpenAI’s “fast mode” doubles standard pricing for the GPT-6 and GPT-5.6 families. The top of the market has not fallen; it has grown a premium shelf.

Open weights set the floor and the floor is uneven. gpt-oss-120b lists at an identical $0.15 and $0.60 on Together, Fireworks and Groq, which is what a commodity looks like. Llama 3.3 70B, by contrast, is $1.04 both ways on Together and $0.10 and $0.32 on DeepInfra’s “Turbo” variant, a tenfold gap for the same weights, which tells you the label on the host page (quantisation, batching, speed tier) matters as much as the model name. Fireworks prices unlisted models by parameter count: $0.10 per million for models under 4B, $0.20 for 4B to 16B, $0.90 above 16B. Our piece on open-weight models closing the gap covers why those weights are good enough to anchor the closed vendors’ pricing.

Figure 1: A price cut rarely arrives evenly across a line-up. Cached and long-context input is where the steepest cuts are now concentrated.

The second axis: caching, batch and off-peak

Headline prices are the first axis. The second axis is the set of discounts that apply to the same model, and in 2026 it moves a real bill more than the headline does.

Caching is the largest single lever

On Anthropic’s page a cache read costs 0.1 times the base input price on most models, 0.05 times on Claude Opus 5.5 and 0.025 times on Claude Fable 5.1. Writing to the 5-minute cache costs 1.25 times input; writing to the 1-hour cache costs 2 times. The arithmetic the page gives is that caching pays off after one read for the 5-minute cache and after two for the 1-hour cache. Minimum cacheable length varies from 512 tokens (Fable 5.1, Opus 5.5, Opus 5) through 1,024 (Sonnet 5) to 4,096 (Haiku 4.5), which means the cheap model has the highest barrier to the discount.

OpenAI’s caching guide says caching is on by default for supported models, that GPT-5.6 and later need a 1,024-token prefix, that a cache read is charged at 0.1 times the uncached rate, and that a cached prefix “remains eligible for reuse for 30 minutes after its most recent write or reuse”. OpenAI’s changelog records extended 24-hour retention launching in November 2025 and becoming the default in May 2026 for earlier models.

Gemini charges both a read price (a tenth of input) and a storage fee: $0.50 per million tokens per hour on 3.8 Flash, $4.50 per million tokens per hour on 3.1 Pro Preview and 2.5 Pro. That storage line makes Gemini caching a different decision from the others: it is cheap for a hot shared prefix and expensive for a cold one.

DeepSeek is the outlier on the low side. Its cache-hit price on V4.1 Flash is $0.006 against a $0.30 cache miss at peak, a 98 percent discount.

Batch halves everything, and it stacks

OpenAI’s Batch API offers a “50% cost discount compared to synchronous APIs”, completes “within 24 hours (and often more quickly)”, and caps a batch at 50,000 requests and 200 MB. Anthropic’s Message Batches API charges “50% of the standard API prices”, says “most batches finishing in less than 1 hour”, expires anything unfinished at 24 hours, and limits a batch to 100,000 requests or 256 MB. Anthropic’s documentation is explicit that “the pricing discounts from prompt caching and Message Batches can stack”, with batch cache hit rates of 30 to 98 percent depending on traffic pattern. Gemini’s pricing page lists batch input and output at half the standard price. Mistral’s says “half price”. Fireworks bills batch “at 50% of serverless pricing”. Together says “up to 50% off serverless rates” with a 24-hour maximum window that it calls a best-effort target.

The practical consequence: a non-interactive workload on a mid-tier model with a well-structured cacheable prefix is paying roughly 5 percent of headline input price on the cached portion and 50 percent on output. Almost no headline cut in the last year has been that large.

Off-peak is a new axis

DeepSeek replaced flat pricing with peak and off-peak rates in August 2026. Its pricing page now states that “Off-peak rates are half of the peak rates. Peak hours are 01:00 - 04:00 and 06:00 - 10:00 UTC, Monday through Friday, excluding Chinese public holidays.” For a European or American team the off-peak window covers most of the working day, which makes the effective price for V4 Pro $0.66 and $1.98 rather than $1.32 and $3.96. It is also the clearest sign yet that a vendor’s serving capacity, not its model, is what it is pricing.

The hidden multipliers: reasoning, tokenisers and long context

A per-token price is only half of a bill. The other half is how many tokens a task consumes, and three things push that number up without touching the price list.

Reasoning tokens are billed as output. OpenAI’s reasoning guide states that reasoning tokens “are not visible via the API” but “still occupy space in the model’s context window and are billed as output tokens”, and that higher effort settings mean the model “thinks more completely” at the cost of more tokens. Google’s pricing page marks every Gemini model’s output price as including thinking tokens. Output is the expensive side of every price in the table (five times input on most rows), so a reasoning setting that doubles output tokens costs more than a 50 percent list-price cut saves. Artificial Analysis, whose index is the one Google cites for token efficiency, reports a cost-per-task figure precisely because “models that produce longer answers or more reasoning tokens will have a higher cost per task, even at identical per-token prices”. Its blended price assumes a 7:2:1 ratio of cache hit, input and output tokens, which is a reasonable picture of an agentic workload and a poor one for a chatbot. We covered why labs moved spend to inference in how test-time compute changed the roadmap.

Tokenisers change under you. Anthropic’s pricing page notes that Claude 4.7 and later models use a newer tokeniser that “produces approximately 30% more tokens for the same text”, with the exact figure depending on the workload. A team migrating from Sonnet 4.6 at $3 and $15 to Sonnet 5 at $2 and $10 sees a 33 percent list-price cut and a roughly 30 percent token-count increase on the same prompts. The per-request bill barely moves. This is not a criticism of the tokeniser, which Anthropic says contributes to the model’s quality, but it is the kind of thing a per-token comparison hides completely.

Long context carries a surcharge on two of three major vendors. Gemini 3.1 Pro Preview is $2 input and $12 output up to 200k tokens and $4 and $18 above it. xAI’s Grok 4.7 doubles from $2 and $6 to $4 and $12 at 200k. Anthropic’s page says Claude 4.6 and later include the full 1M-token window at standard pricing, “A 900k-token request is billed at the same per-token rate as a 9k-token request.” Retrieval-heavy products that regularly cross 200k tokens should treat that difference as part of the price.

Then there are the multipliers on top. Anthropic charges 1.1 times for US-only inference geography on Claude 4.6 and later models. Regional endpoints on Bedrock and Vertex carry a 10 percent premium over global ones. Fast mode doubles the price on both OpenAI and Anthropic. Anthropic’s tool-use system prompt adds 286 to 675 tokens per request depending on model, and declaring its browser toolset adds about 6,600 input tokens before any work is done. None of these are large individually. Together they explain why the bill of a product that “just upgraded” does not look like the press release.

Figure 2: Four steps multiply tokens or price before the two discounts divide it. Output tokens are the line that reasoning effort inflates and no list-price cut has touched.

Why prices fall

Five forces push list prices down, and all five are still operating.

Hardware. NVIDIA’s Rubin announcement on 5 January 2026 claims “up to 10x reduction in inference token cost compared to Blackwell platform” for mixture-of-experts models, with 50 petaflops of NVFP4 inference compute per GPU and partner availability in the second half of 2026. The cloud rental market shows what the previous generation did: as of 27 September 2026 Lambda lists an H100 SXM at $3.99 per GPU-hour in an 8-GPU instance and a B200 at $6.69; DeepInfra lists an H100 at $2.20, an H200 at $2.69 and a B200 at $3.69 per hour; Fireworks lists on-demand H100 at $8.00 and B200 at $13.00. The spread across hosts (nearly four times for the same H100) is the same lesson as the Llama 3.3 70B row above: the vendor’s utilisation and margin matter as much as the silicon.

Serving optimisations. The a16z post lists the software side: quantisation from 16-bit to 4-bit weights, better batching, and smaller models that hit the same score. NVIDIA’s 4-bit NVFP4 figure above is the hardware end of the same trend. Mixture-of-experts architectures reduce active compute per token; DeepSeek’s V4.1-Flash release note describes a 552B-parameter model that activates 8B parameters per input token and 16B per output token, which is why a model that large can list at $0.30 and $1.20.

Distillation and small-model progress. The clearest evidence is the price table itself. The task that needed a flagship in 2024 runs on a mid-tier model in 2026 at a fifth of the price, and Google’s changelog shows three successive Flash models launching at the same price with better scores each time.

Open-weight competition. Every closed vendor now has a floor set by hosts serving open weights at cost. When gpt-oss-120b is $0.15 and $0.60 on three hosts, a closed small model cannot charge $1 and $5 for a comparable score indefinitely, and the July and September cuts to Luna suggest OpenAI knows it.

Subsidy. Introductory pricing is a subsidy with a published expiry date. Google’s Flash intro prices double on 1 January 2027. Anthropic’s Sonnet 5 price was introductory until the company decided it was not. DeepSeek’s V4 Pro carried a promotional discount before the August move to peak pricing. Some fraction of the decline in the table is vendors buying share, and that fraction can be withdrawn.

Who keeps the saving, and who does not

The same cut produces different outcomes depending on how a product spends tokens.

High-volume, simple, fixed workloads keep all of it. A classifier, an extraction pipeline, a nightly summarisation job: the token count per task is stable, the quality bar is known, and the work can run in batch. A team that moved this kind of workload from GPT-5.6 Luna at its July launch price to GPT-6 Luna in batch today pays $0.05 per million input tokens instead of $1, twenty times less, for a job that has not changed.

Token-dominated agents keep some of it, if they choose to. An agent whose cost is dominated by tokens rather than by tool calls, sandboxes or human review sees its per-step cost fall. But the same team is under pressure to raise the step budget, turn reasoning effort up and lengthen memory, because each of those improves the success rate. Anthropic’s pricing page separates session runtime ($0.08 per session-hour for managed agents) from tokens, which is a useful reminder that the token line is not the only one. Our real cost of an AI product piece goes through the rest of that bill.

Model-agnostic architectures capture the most. The cuts of the last twelve weeks landed on specific models: Luna twice, Sonnet 5 by cancellation, the Flash line by replacement. A product whose model choice is a configuration value behind a gateway can move to whichever row of the table is cheapest this quarter; one with prompts, evaluations and tool schemas welded to a single provider cannot. Routing traffic through a gateway such as Bifrost, the open-source Go AI gateway from Maxim AI, lets a team swap the model behind a route without touching application code; our comparison of LLM gateways covers what else that layer should do.

Latency-bound products keep little. The cheapest rows in the table are batch and off-peak. A product that has to answer in under a second cannot use either, cannot always wait for a cache write to pay back, and often ends up on fast mode at double price. For these products the relevant trend is tokens per second, not dollars per token.

Reasoning-heavy workloads keep the least. When output tokens dominate and effort settings are high, the bill tracks the output price, which is the slowest-moving number on every vendor’s page. Fable 5.1 output is $50, Astra output is $50, and a “pro” tier reaches $180 or $600. A hard reasoning task that needs the top tier has seen no cut at all.

Teams locked to one provider keep only what that provider chooses to give. Anthropic’s Haiku 4.5 customers have had no cut this year while OpenAI’s Luna customers have had two. That is not a judgement on either vendor; it is the point. The saving is distributed by competition, and a team that cannot move is not part of the competition.

Figure 3: The same price cut produces three different outcomes depending on how the product spends tokens.

The counter-argument: total bills are rising anyway

The strongest case against the “prices are falling” story is that almost nobody’s bill is.

Usage per task is rising faster than price per token is falling. Google’s own framing of the 3.6 Flash launch was cost per agentic task, not per token, because the number of tokens an agent burns per task has been climbing with every step-budget increase. Artificial Analysis moved to cost-per-task reporting for the same reason. If a team’s tokens per task triple in a year and the price halves, the bill rises 50 percent, and that is a common pattern for agentic products. The counter is real but it is a choice: the tokens are being spent because the extra quality is worth it, and a team that wanted the old quality at the new price could have it.

There is a hardware floor and it is not zero. At DeepInfra’s $2.20 per H100-hour, a host has to move a great many tokens per hour to list a 70B model at $0.10 per million input tokens and still cover power, networking and margin. The Llama 3.3 70B gap between Together ($1.04) and DeepInfra ($0.10) is partly quantisation and partly the fact that one of those prices is much closer to the floor than the other. Rubin’s promised 10 times will lower the floor again in the second half of 2026, but the promise is NVIDIA’s, for MoE workloads, and cloud list prices for the previous generation have not fallen ten times. Epoch’s caution that the fastest declines “have occurred in the past year” and may not persist belongs here.

Premium tiers are absorbing the demand that matters. Every cut in the table sits below a tier that did not move. Vendors have found that the customers with the hardest problems will pay $50, $180 or $600 per million output tokens, and they have built products at those prices. The falling price of the median token is compatible with a rising price of the marginal one.

Some prices are going up. The Gemini Flash intro rates double on 1 January 2027. DeepSeek’s move to peak pricing doubled its daytime rate for anyone who was on the promotional price. Anthropic scheduled a 50 percent rise on Sonnet 5 before cancelling it. A planning assumption that every line item falls every year is wrong on the evidence of this summer alone.

None of this reverses the argument. It narrows it. Constant-capability inference is getting cheaper very fast; the bill of a product that keeps raising its capability target is not, and the gap between the two is a product decision rather than a market fact.

A worked example: one support assistant at three price points

Consider a customer-support assistant that handles 50,000 conversations a day. Each conversation sends 20,000 input tokens across its turns, of which 15,000 are a stable prefix (system prompt, tool schemas, policy documents) that can be cached, and 5,000 are conversation-specific. It produces 2,500 output tokens. The numbers are illustrative; the prices are the vendor list prices in the table, read on 27 September 2026, and the arithmetic is per conversation.

Four things stand out.

First, caching is worth more than a tier change at the top. Turning caching on for the frontier model saves $85,500 a month; moving from frontier-with-caching to mid-with-caching saves $52,500. The cache is the bigger lever and it requires no quality re-evaluation.

Second, reasoning effort can erase a tier. The mid-tier model at an assumed 10,000 output tokens per conversation costs more than the frontier model at 2,500. The 4 times output figure is an assumption, not a measurement, and the right number for a given product has to come from its own logs. But the direction is not in doubt: OpenAI and Google both bill thinking as output, and output is the $10 to $50 line.

Third, the small tier is not a discount, it is a different business. At $2,850 a month the model cost of this product is a rounding error next to the engineers running it, which is the position every fixed-workload product should try to reach. The only question is whether GPT-6 Luna passes the product’s evaluation set, and that question costs an afternoon to answer.

Fourth, the open-weight row is competitive with the closed small tier but not clearly cheaper, and it is only this cheap off-peak. Open weights set the floor; they do not always sit on it.

What to do

The decline is real, it is uneven, and it rewards teams that are set up to move. Concretely:

1. Re-benchmark quarterly, against the cheapest tier first. Keep an evaluation set per production use case and run it against the current small and mid tiers every quarter. The model that passed the bar at a tenth of the price is the single largest saving available and it costs nothing to find. This summer alone produced a new small model at a tenth of the July price and a new mid model at half.
2. Route by task, not by product. Classification, extraction and formatting belong on the small tier or open weights. Multi-step agent work belongs on the mid tier with a capped reasoning budget. Reserve the frontier and premium tiers for the tasks that fail everywhere else, and measure that they fail. A gateway that makes the model a configuration value is what makes this cheap to change.
3. Design for the cache. Put the stable content (system prompt, tool schemas, reference documents) first and byte-identical in every request. Check the minimum prefix length for the model (512 to 4,096 tokens across the Anthropic line, 1,024 on GPT-5.6 and later). Use the 1-hour cache for batch jobs. On Gemini, watch the storage fee for cold prefixes.
4. Move everything that can wait into batch. Backfills, evaluations, nightly jobs and any pipeline with a human at the end of it should run at half price with a 24-hour window. On Anthropic the batch discount stacks with caching.
5. Budget in dollars per task and log the output tokens. Token prices are falling; tokens per task are rising; the product metric is the ratio. Log reasoning and output tokens per task separately and set a budget per route. A reasoning-effort change is a pricing change and should go through the same review.
6. Negotiate committed spend on a short horizon, with a price-match clause. Anthropic’s page says volume discounts are “negotiated on a case-by-case basis”; OpenAI, Google and the open-weight hosts have equivalent programmes. A commitment priced on today’s list rates for three years is a bet against the clearest trend in the industry. Commit for a year at most, and ask for language that passes list-price cuts through.
7. Track the increases as carefully as the cuts. Put 1 January 2027 in the calendar for Gemini Flash. Check whether a workload runs in DeepSeek’s peak window. Read the footnotes on every launch for the word “introductory”.

Prices will keep falling. The forces behind the decline are hardware, software, distillation and competition, and none of them has run out. But the saving is not delivered to a bill; it is delivered to a market, and a team collects its share only by being able to move within that market. The products that will be cheap to run in 2027 are not the ones on the cheapest model today. They are the ones for which changing the model is a Tuesday.