The real cost of an AI product, line by line: tokens are rarely the biggest line

Ask a team what their AI feature costs and they will quote a token bill. It is the number that arrives monthly, it is the one finance asks about, and it is the one the falling price of inference keeps improving. It is also, in the cost models we have built and the survey data that exists, usually not the largest line. ICONIQ’s December 2025 survey of 202 companies building AI products found model inference at 20 percent of AI product cost before launch and 23 percent once scaling, with talent at 32 and 26 percent respectively. The rest is infrastructure and cloud, data, training and compliance.

This explainer is the full ledger. For each line we describe the mechanism that generates the cost and give a list price from a page we read in September 2026, then assemble a complete monthly model for one hypothetical product at three volumes. Every unit price is sourced; the assumptions about volume are ours and are labelled as such. The aim is not the totals, which will be wrong for your product, but a cost model with enough lines that “why did the bill go up” becomes a lookup rather than an investigation.

Line 1: model inference

The inference line is four prices, not one. As of September 2026 every major provider bills input tokens and output tokens separately, bills cached input at a steep discount, and bills reasoning at the output rate.

Input and output. Output is consistently the expensive direction. On Anthropic’s price list, Claude Sonnet 5 is $2 per million input tokens and $10 per million output; Opus 5.5 is $4 and $20; Fable 5.1 is $10 and $50; Haiku 4.5 is $1 and $5. OpenAI’s GPT-6 line runs from gpt-6-luna at $0.10 in and $0.50 out, through gpt-6-sol at $2 and $10, to gpt-6-astra at $10 and $50. Google’s Gemini 3.1 Pro Preview is $2 in and $12 out for prompts up to 200,000 tokens and $4 and $18 above that; Gemini 3.8 Flash is $0.75 and $3.75. Mistral Large 3 is $0.50 and $1.50. DeepSeek’s V4 Pro is $1.32 in and $3.96 out at peak, and half that off-peak. The ratio of output to input is five to one on almost every current list, which is why output limits matter more than prompt trimming for chat products that generate long answers.

Reasoning tokens. Models that think before answering bill the thinking. OpenAI’s reasoning guide states that reasoning tokens “are not visible via the API” but “are billed as output tokens”, and recommends reserving at least 25,000 tokens for reasoning and output when starting out. Google’s Gemini price list labels its output rate as including thinking tokens. Anthropic’s Claude Code documentation says thinking tokens are billed as output and that the default budget “can be tens of thousands of tokens per request depending on the model”. A reasoning-heavy task can therefore cost more in invisible output than in the answer the user sees, a dynamic we covered in how test-time compute changed the roadmap.

Caching. Caching is the single largest discount on the price list and the most misunderstood. On the Claude API, a cache read costs 0.1x the base input price on most models, 0.05x on Opus 5.5 and 0.025x on Fable 5.1; a 5-minute cache write costs 1.25x and a 1-hour write 2x. Anthropic’s pricing page spells out the consequence: caching “pays off after one cache read for the 5-minute duration”. The cache is a prefix cache, created in the order tools, then system, then messages, so any change to the tools block invalidates everything after it. Minimum cacheable lengths run from 512 tokens on Fable 5.1 to 4,096 on Haiku 4.5; shorter prompts marked for caching are silently not cached. OpenAI’s caching is automatic, requires 1,024 tokens on GPT-5.6 and later, retains the prefix for 30 minutes after its last use, and bills cached reads at 0.1x. Gemini charges both a per-token cache rate ($0.20 per million on Gemini 3.1 Pro Preview) and storage at $4.50 per million tokens per hour, which is a cost line most estimates forget. DeepSeek and Mistral both publish cache-hit rates at roughly a tenth of the miss rate.

Batch. Every provider on this list offers 50 percent off for asynchronous work. OpenAI’s batch API completes “within 24 hours (and often more quickly)” and accepts up to 50,000 requests per file. Anthropic’s Message Batches API is limited to 100,000 requests or 256 MB, most batches finish within an hour, and the documentation confirms that batch and caching discounts stack. Gemini and Mistral each state a 50 percent batch discount. For evaluation suites, re-indexing and any overnight pipeline this is the cheapest price on the page.

Prices per million tokens from each vendor’s pricing page, read September 2026. Gemini 3.8 Flash is scheduled to rise to $1.50 in and $7.50 out on 1 January 2027, a reminder that list prices do not only fall.

Two footnotes on the price list matter for budgeting. First, prices can go up: Google’s page states that Gemini 3.7 and 3.8 Flash double on 1 January 2027, and Anthropic’s Sonnet 5 price was announced as introductory before the scheduled increase was cancelled. Second, tokenisers change: Anthropic notes that Claude 4.7 and later use a tokeniser that “produces approximately 30% more tokens for the same text”, so a model swap can raise the bill at a constant price per token.

Line 2: retrieval

Anything that assembles the prompt costs money before the model sees it: embedding the corpus, storing the vectors, querying them, and reranking the candidates. Each has its own unit.

Embeddings. Embedding is cheap per token and expensive per re-index. OpenAI’s text-embedding-3-small is $0.02 per million tokens and text-embedding-3-large $0.13; Voyage’s voyage-3.5 is $0.06 and voyage-3-large $0.18; Mistral Embed is $0.10; Google’s Gemini Embedding 2 is $0.20 for text. Embedding a 200-million-token corpus once costs $4 to $40 at those rates. The cost that hurts is the recurrence: every change of embedding model or chunking strategy re-runs the whole corpus, and the old index cannot be queried by the new model.

Vector storage. Pinecone’s serverless pricing, on its Standard plan, is $0.33 per GB per month for storage, $4 to $4.50 per million write units and $16 to $18 per million read units, with a $50 monthly minimum. Its cost documentation says a query “uses 1 RU for every 1 GB of namespace size, with a minimum of 0.25 RUs per query”, and that top_k and metadata flags do not change the price. A 3 GB namespace therefore costs about 3 read units, or $0.00005, per query. pgvector on a managed Postgres has a different shape: Neon’s Launch plan charges $0.106 per compute-unit hour and $0.35 per GB-month, with pgvector included in every plan, so an always-on single compute unit is about $77 a month before storage. Qdrant Cloud publishes a free tier (0.5 vCPU, 1 GB RAM, 4 GB disk) and bills paid clusters hourly on vCPU, memory, disk and backups, but its pricing page gives no unit rates; you need the calculator or a sales call.

Retrieval unit prices from each vendor’s pricing page. Qdrant Cloud publishes a free tier but no paid unit rates.

Reranking. Rerankers are priced per request or per token. Pinecone’s hosted reranking is $2 per 1,000 requests. Voyage’s rerank-2.5 is $0.05 per million tokens and rerank-2.5-lite $0.02, so reranking 20 candidates of 300 tokens each costs $0.0003 per query on the full model. Cohere’s public pricing page defines a search unit as “one query with up to 100 documents” but, as of September 2026, does not print a per-unit price. In our worked model reranking is the largest retrieval line, ahead of storage and reads combined.

Line 3: agent loops

An agent is a loop that sends the conversation back to the model after every tool call. The token cost of a task is therefore not the size of the answer but the size of the context multiplied by the number of steps, and the number of steps is not known in advance.

The documented examples are sobering. Anthropic’s January 2025 engineering post on SWE-bench Verified notes that “many successful runs took hundreds of turns for the model to resolve, and >100k tokens”, with the agent sampling until it either finishes or exceeds its 200,000-token context. Epoch AI’s SWE-bench Verified evaluation, updated in March 2026, caps each task at “2M uncached read/write tokens (including reasoning), and 20M cached token reads”, a ceiling that only makes sense because real runs approach it. Anthropic’s Claude Code cost documentation gives the production figure: across enterprise deployments “the average cost is around $13 per developer per active day and $150-250 per developer per month, with costs remaining below $30 per active day for 90% of users”. The same page states that agent teams, where several instances coordinate, “use approximately 7x more tokens than standard sessions”.

The mechanism behind these numbers is the tool result. Every result returned to the model becomes input on every subsequent turn. Anthropic’s pricing page lists the fixed overheads: a tool-use system prompt of 286 to 675 tokens depending on model, 700 tokens for the text editor tool, about 4,500 tokens to declare the computer-use toolset and 6,600 for browser use, plus screenshots billed as images. Web search is a separate $10 per 1,000 searches. Caching absorbs most of the repetition, which is why agent products live or die on cache hit rate, but cache writes still cost 1.25x on every new tool result, and a break longer than the cache lifetime reprocesses the entire context at full price.

Two consequences for the cost model. First, track the distribution of cost per task, not the mean: the mean is set by a small number of runaway runs. Second, the fixes that matter are structural, not per-token: hard step caps, tool results trimmed before they enter context, and cheaper models for sub-tasks. Our survey of agent frameworks found that few frameworks expose these controls by default.

Line 4: evaluation

Evaluation is a recurring cost, not a launch cost, because every prompt edit, model swap, retrieval change and provider price cut is a change to the system that needs checking before it ships. Teams that skip it pay for it in the human review line instead.

It costs tokens and it costs people. The token part has two shapes. Online evaluation samples live traffic and scores it with a judge model; the cost is the judge’s token price times the sample rate. A Haiku 4.5 judge reading a 5,000-token transcript and writing a 300-token verdict costs $0.005 plus $0.0015, or $0.0065 per judged conversation at September 2026 list prices. Offline evaluation re-runs a fixed regression suite on every release; because it is not latency-sensitive it belongs on a batch API at half price, and Anthropic’s batch documentation specifically recommends the 1-hour cache for suites that share context. The people part, building the test set and periodically checking the judge against human labels, is engineering time and appears in line 8.

The evidence that evaluation is under-resourced comes from the same ICONIQ survey: only 52 percent of builders reported adopting automated evaluation frameworks, with the rest relying on user feedback and manual checks. That is a cost too; it just lands on a different line.

Line 5: observability

Storing prompts, completions, tool calls and traces at production volume is a real storage and processing cost, and the tooling to search them is either a vendor bill or engineering time. Public pricing is available for at least one open-source option. Langfuse Cloud charges $29 a month on its Core plan with 100,000 units included and 90 days of retention, $199 on Pro with three years of retention, and overage on a sliding scale from $8 per 100,000 units down to $6 at volume. A unit is “any tracing data point you send”, so one four-turn conversation with retrieval spans and a score is around 15 units, and a million conversations a month is around 15 million units, or roughly $1,200 a month on the published scale. Maxim AI, whose gateway we discuss below, does not publish per-unit prices for its evaluation and observability platform as of September 2026; its pricing page lists the gateway’s open-source tier as free and enterprise as custom.

The lever here is sampling. Full-fidelity traces for the first weeks of a feature, then sampled traces with full retention only for failures and low scores, keeps the line flat while the product grows.

Line 6: the gateway and infrastructure layer

Between the application and the providers sits a gateway that handles keys, routing, failover, budgets and logging. It is either a hosted service billed per request or a self-hosted process that costs a VM and some attention. We compared the options in five LLM gateways compared; the cost question is simpler than the feature question.

Bifrost, the open-source Go gateway from Maxim AI, is the self-hosted example. Its repository is Apache 2.0 licensed, its documentation lists over twenty providers behind one OpenAI-compatible API, and it runs from npx -y @maximhq/bifrost or a Docker image on port 8080. The published benchmark claims 11 µs of added overhead per request at 5,000 requests per second on a t3.xlarge instance; that is a vendor figure, not independently measured. The open-source tier includes semantic caching, virtual keys with budgets and rate limits, and Prometheus and OpenTelemetry export; guardrails, clustering, SSO and audit logs are in the paid enterprise tier. The cost of running it is the cost of the instance: a t3.xlarge is $0.166 an hour on demand, about $121 a month, per the AWS list price as mirrored by instances.vantage.sh.

The point is that the gateway line is small in either direction. What it changes is the visibility of every other line: per-key budgets and per-request cost attribution are what make the rest of this ledger measurable.

Figure 1: Costs that scale with usage and costs that scale with change need different controls.

Line 7: human review and labelling

Any feature with a quality bar has humans behind it: reviewers checking a sample of outputs, support staff handling escalations when the model is wrong, and domain experts labelling data for the evaluation set. This line is invisible in the cloud bill and, in regulated domains, is often the largest single cost. Andreessen Horowitz’s 2020 analysis of AI business models, still the most cited primary treatment, put human-in-the-loop costs at 10 to 15 percent of revenue for some companies and observed that the two cost centres are linked: “reducing one tends to drive an increase in the other”.

The tooling is cheap and the people are not. Amazon’s Augmented AI charges $0.03 per human-reviewed object for the first 100,000 a month, then $0.02. Mechanical Turk takes a 20 percent commission on worker rewards, rising to 40 percent for tasks with ten or more assignments. The expensive part is the reviewer’s time. The US Bureau of Labor Statistics reports a median annual wage of $104,300 for software quality assurance analysts and testers in May 2025, or about $50 an hour; a five-minute review at that rate costs about $4.20 before employer on-costs, one hundred times the tooling fee.

Line 8: people

The largest line before launch, according to ICONIQ, is talent: 32 percent of AI product cost pre-launch, 30 in beta, 28 at general availability and 26 at scale. That is prompt engineering, evaluation engineering, retrieval tuning, and the recurring work of model swaps. Every deprecation notice, price cut and better model triggers a cycle of re-prompting, re-evaluating and re-tuning, and a team that swaps models quarterly to capture price cuts needs to budget the swaps.

The unit price is a salary. The BLS median for software developers was $135,980 in May 2025, roughly $11,300 a month before benefits and employer taxes. Half an engineer on a small feature is $5,700 a month; two engineers on a large one is $22,700. For most products below a million interactions a month, this line alone exceeds the token bill.

Line 9: self-hosting GPUs versus the API

At some volume every team asks whether to rent GPUs and serve an open-weight model. The arithmetic has three inputs: the GPU hour, the throughput a serving stack achieves, and the utilisation you can sustain.

GPU hours. AWS’s EC2 Capacity Blocks price a p5.48xlarge (eight H100s) at $41.528 an hour in US regions, or $5.191 per GPU-hour; a p5e (H200) at $47.76, or $5.97 per GPU; and a p6-b200 at $98.84, or $12.355 per GPU. Lambda’s on-demand list is $3.99 per GPU-hour for H100 SXM in an 8-GPU node, $4.29 for a single H100, and $6.69 for B200; its reserved one-click clusters are higher, at $6.16 per H100-hour for 16 GPUs. All figures are September 2026 list prices excluding tax.

Per-GPU-hour list prices from the AWS EC2 Capacity Blocks and Lambda pricing pages, excluding tax and OS charges.

Throughput. An independent benchmark published by the GPU cloud Spheron in 2026 ran Llama 3.3 70B Instruct in FP8 on a single H100 SXM5 80 GB with 512-token inputs and 256-token outputs. At 100 concurrent requests it measured 2,400 output tokens a second on vLLM 0.18, 2,460 on SGLang 0.5.9 and 2,780 on TensorRT-LLM 1.2; at a single request, 120 to 130 tokens a second. Taking the vLLM figure, one fully loaded H100 produces about 8.6 million output tokens an hour and processes about 17.3 million input tokens in the same period.

Break-even. The closest hosted comparator in class is Mistral Large 3 at $0.50 per million input and $1.50 per million output. The API price of an hour of that H100’s output is 17.3 million input tokens at $0.50 plus 8.6 million output tokens at $1.50, about $21.60. The GPU costs $3.99 to $5.19 an hour. Self-hosting therefore breaks even at roughly 18 to 24 percent sustained utilisation, and wins at full utilisation by a factor of four to five.

Three things erode that. Real traffic is bursty and a single GPU is not a production deployment, so the practical utilisation of a two-GPU, always-on setup is often below the break-even point. The comparison against a frontier API is not like for like: Sonnet 5 output is $10 per million, but nobody self-hosts Sonnet 5. And the engineering line grows: someone has to own the serving stack, the upgrades and the on-call rota, at the salary rates in line 8. Our judgement is that self-hosting is a volume decision first and a data-residency decision second, and rarely a cost decision below several hundred million tokens a day.

Figure 2: The API converts fixed costs into variable ones; self-hosting reverses the trade and adds an engineering line.

Line 10: compliance and security

Data residency, regional routing and government-grade endpoints carry explicit surcharges. Anthropic bills US-only inference at 1.1x on all token categories for Claude 4.6 and later models. OpenAI charges “a 10% uplift” for FedRAMP endpoints and a 10 percent uplift for data residency on models released on or after 5 March 2026. Anthropic’s cloud pricing notes that regional and multi-region endpoints on Bedrock and Google Cloud carry a 10 percent premium over global endpoints. ICONIQ’s respondents put compliance at 5 to 6 percent of AI product cost across stages, with model training at 11 to 13 percent and data storage and processing at 14 to 15 percent.

The rule of thumb: if you must pin inference to a jurisdiction, add 10 percent to line 1 and check whether the same requirement forces the vector store and the trace store into the same region, because the cheaper serverless tiers are often not available there.

A worked example: one product, three volumes

The product is a customer-support assistant with retrieval over a 100,000-document help corpus. Each conversation is four turns. The main model is Claude Sonnet 5. Our assumptions per conversation: a 3,000-token system prompt and tool block that is cached; 2,000 tokens of retrieved context plus 200 tokens of user text per turn, uncached; 300 output tokens per turn; conversation history served from cache. That produces, per conversation, about 10,000 cache-write tokens, 25,000 cache-read tokens, 9,000 uncached input tokens and 1,200 output tokens. Retrieval is four Pinecone queries against a 3 GB namespace and four hosted reranks. Observability is Langfuse Cloud at 15 units per conversation. Evaluation is a 5 percent online sample judged by Haiku 4.5 plus a 500-case regression suite run eight times a month through the batch API. Human review samples 1 percent of conversations at five minutes each. Engineering is 0.5, 1 and 2 developers at the BLS median. All prices are September 2026 list prices from the sources above; salaries exclude on-costs.

Monthly cost model. Per-conversation inference is $0.060: $0.025 cache writes, $0.005 cache reads, $0.018 uncached input, $0.012 output. Corpus embedding ($4 to $26 one-off at OpenAI rates) and the initial Pinecone upsert (about $14) are excluded as one-off costs. US-only inference would add 10 percent to the inference line.

Three things stand out. At 10,000 conversations a month the token bill is a rounding error next to the engineer. At 100,000 the human review line is already two-thirds of inference, and it is the line that grows fastest if quality slips, because escalations are reviews too. At a million, inference is finally the largest line, at 44 percent, which is close to where the old version of this article guessed it would be and close to ICONIQ’s finding that inference becomes “the dominant cost driver at scale”. A team at that volume would also cut the review sample to 0.2 percent, taking that line to about $8,400, which pushes inference above half. The ledger changes shape as the product grows, and a model with one line cannot show that.

The levers that move the total most

Each lever moves one line. Knowing which is the point of the ledger.

Caching the prefix. On Sonnet 5 the 3,000-token system prompt, sent on each of four turns, costs $24 per thousand conversations uncached and about $2.40 once the cache is warm and shared across conversations. At a million conversations that is the difference between $24,000 and $2,400 a month on the prefix alone. The mechanics matter: the breakpoint goes on the last block whose prefix is identical across requests, tool definitions must not change between calls, and a prompt below the model’s minimum cacheable length will silently pay full price.

Routing by task. Haiku 4.5 is half the price of Sonnet 5 on every category, and gpt-6-luna is a twentieth of gpt-6-sol. If 60 percent of turns in the worked example are classification, extraction or short factual replies that a small model handles, per-conversation inference falls from $0.060 to about $0.042, a 30 percent cut on the largest variable line. The route decision has to be evaluated, which is why line 4 exists.

Prompt length discipline. Uncached context is the line that grows silently. In the worked example every extra 1,000 tokens of retrieved context per turn adds 4 million tokens per thousand conversations, or $8,000 a month at a million conversations on Sonnet 5’s input price. Retrieving five chunks instead of eight is worth more than most model swaps.

Output limits and reasoning effort. Output is five times the input price on nearly every list. Setting max_tokens to what the interface can show, and setting reasoning effort to the lowest level that passes the evaluation suite, bounds the line that runaway generations blow through. OpenAI’s guide notes that max_output_tokens caps reasoning and visible output together.

Batch for everything asynchronous. The 50 percent discount applies to evaluation suites, re-indexing, nightly summarisation and backfills on every provider on our table, and stacks with caching on Anthropic. DeepSeek adds a further halving off-peak. Nothing that can wait an hour should run at the synchronous price.

None of these levers moves the review, evaluation or engineering lines. A 90 percent cut in token spend at 100,000 conversations a month saves $5,400 against a $23,000 total. The way to move the human lines is measured quality: a review rate that falls as evaluation scores rise, and a change budget that shrinks as the evaluation suite gets faster to run.

What margins look like

The public record on AI product margins is thin but improving. Andreessen Horowitz’s February 2020 essay set the baseline: AI company gross margins of 50 to 60 percent against 60 to 80 percent or more for software, with cloud costs of 25 percent or more of revenue and human labelling and review on top. Sequoia’s June 2024 “AI’s $600B question” approached the problem from the supply side, doubling Nvidia’s revenue run-rate for data-centre costs and doubling it again for a 50 percent end-user gross margin, and asked where the revenue would come from.

The 2026 data suggests the answer is arriving slowly. ICONIQ’s January 2026 snapshot, from a December 2025 survey with 269 respondents on this question, reports average gross margin on AI products of 41 percent in 2024, 45 percent in 2025 and a projected 52 percent in 2026, with companies pursuing “balanced differentiation” highest at 53 percent and application-layer-only companies lowest at 45. Its July 2026 update, from about 300 executives, puts 2025 at 45 percent and projects 53 percent for 2026 and 59 percent for 2027; the January snapshot attributes the trend to model selection, routing strategies and infrastructure efficiency, “not simply pricing power”.

Two public companies have put numbers on individual products. Wix’s Form 6-K of 4 August 2026 says it expects Base44, its app-building product, “to achieve ~60% non-GAAP gross margin in 2H, a significant improvement from near-zero non-GAAP gross margin entering the year”, credits the deployment of its own Base 1 model as “a turning point in lowering our AI inference and compute costs”, and guides to about two points of consolidated gross margin improvement in the second half. Doximity, in its fiscal third-quarter 2026 prepared remarks, kept its annual outlook flat despite a revenue beat, citing lower fourth-quarter revenue expectations and “higher AI infrastructure investment driven by a strong increase in usage”; it reported non-GAAP gross margin of 91 percent against 93 percent a year earlier, and committed to maintaining 50 percent or better adjusted EBITDA margins while continuing to invest in AI infrastructure.

Against that, the deflation is real. Andreessen Horowitz’s November 2024 “LLMflation” note found that for constant capability the price of inference fell about tenfold a year, from $60 per million tokens for GPT-3-level performance in 2021 to $0.06 three years later. Epoch AI’s March 2025 analysis across six benchmarks found declines of 9x to 900x a year depending on the task, with GPT-4-level performance on PhD-level science questions falling 40x a year. The catch, visible in every statement above, is that usage grows faster than prices fall: Doximity’s AI cost rose with “a strong increase in usage”, and Wix’s margin fix was a cheaper model, not a cheaper price list.

Judgement, and a checklist

Our judgement is that the token bill is the easiest line to measure and the easiest to cut, and that both facts make it a poor proxy for the cost of an AI product. Below a million interactions a month the ledger is dominated by people: engineers changing the system and reviewers checking it. Above that, inference takes the lead, but only if review has been reduced by measured quality rather than by neglect. The survey data, the public company statements and our own worked model all point the same way.

Build the ledger before the bill arrives. In order:

1. Instrument cost per completed task, split into cached input, uncached input, output and reasoning, per model. Cost per token hides which lever to pull.
2. Set a cache hit-rate target and alarm on it. A change to tool definitions that silently invalidates the prefix cache is a price increase of up to 10x on that line.
3. Cap steps and output for agents, and report the 95th percentile of cost per task, not the mean.
4. Give evaluation two budget lines: a recurring online sample and a fixed per-release suite on the batch API.
5. Put human review on the ledger at the reviewer’s real hourly cost, and set a target for the sample rate to fall as scores rise.
6. Fund a standing model-change budget rather than treating each swap as a project; the price cut is only worth the swap if the swap is cheaper than the saving.
7. Rerun the self-hosting arithmetic quarterly with your own utilisation, not the benchmark’s. The break-even moves every time a GPU price or an API price changes.
8. Add the compliance uplift explicitly if you need regional inference, and check the vector and trace stores for the same constraint.

When the next price cut is announced, the ledger will tell you which line it touches. In most products it will be a line that is a quarter of the total, and the other three-quarters will be waiting.