GenAI Brief

Inference prices keep falling. Here is who actually benefits

GenAI Brief — Fri, 25 Sep 2026 00:00:00 GMT

Every few months a vendor announces that a model "matching last year's frontier" now costs a tenth of what it did. The announcements have become routine enough that it is worth asking a boring question: who actually keeps the money?

The short answer is that the savings are real but they are not evenly distributed. A price cut at the model layer flows through to product margins only when a product's cost structure is token-bound, its volume is fixed and its quality bar is already met. That describes fewer businesses than the headlines imply.

The trend, and why it holds

Tracking published list prices for a fixed capability level, the cost of frontier-class inference has been falling at roughly an order of magnitude every 18 months. Epoch AI's price tracking and a16z's "LLMflation" analysis arrive at similar slopes using different methods. Three mechanisms drive it:

Distillation and small-model progress. Each generation of large models is used to train smaller successors that hit the previous quality level at a fraction of the parameters. The "GPT-4-class" bar has been cleared by progressively smaller models every year.
Serving efficiency. Speculative decoding, better batching, quantisation and purpose-built inference chips all raise tokens-per-dollar without touching the model.
Competition. With several labs shipping comparable capability, list price has become a lever. Open-weight releases anchor the floor, because anyone can serve them at cost.

None of those forces show signs of exhausting, so the sensible planning assumption is that a given capability will cost a third of today's price a year from now.

Where the cuts land first

Cuts do not arrive uniformly across a vendor's line-up. The pattern is consistent enough to plan around:

Tier	Typical behaviour	What to expect
Small / fast	Cut aggressively and often	Near-zero marginal cost; used as a loss leader
Mid-size	Replaced by a better model at the same price	Quality improves, price holds
Flagship	Held at launch price, then superseded	Rarely discounted; deprecated instead
Long context and cached input	Fastest-moving line item today	Expect the largest percentage cuts here

The last row matters most for builders. Output-token prices grab headlines, but for retrieval-heavy and agentic workloads the majority of spend is input tokens, and cached-input pricing is where vendors are now competing hardest.

Figure 1: A price cut rarely arrives evenly across a line-up. Cached and long-context input is where the steepest cuts are now concentrated.

Who benefits

Application builders with fixed workloads capture the savings directly. A support-ticket classifier that handles a set volume of tickets a day sees its bill fall with each cut. These businesses have strong incentive to re-evaluate their model choice every quarter, because yesterday's premium model is now a mid-tier price.

Agentic products often do not. When a model gets cheaper, teams give agents more steps, longer memories and more retries, and the quality improves while the bill stays flat. That is a rational trade, but it means "inference is getting cheaper" does not translate to "our margins are improving". Treat cheaper tokens as a quality budget, not a cost saving, unless you deliberately cap usage.

End users benefit through free tiers and higher limits more than through lower prices. Consumer subscription prices have been remarkably sticky even as the underlying cost fell; vendors pocket the difference or reinvest it in capability.

Cloud providers and inference startups are squeezed. Serving open-weight models is a commodity business with thin margins, and every list-price cut from a lab compresses them further. Expect consolidation.

Figure 2: The same price cut produces three different outcomes depending on how the product spends tokens.

What to do about it

Re-benchmark quarterly. Keep an evaluation set for each production use case and re-run it against the current cheap tier. Downgrading a model that still passes your bar is the single easiest saving available.
Budget in dollars per task, not dollars per token. Token prices are falling; tokens per task are rising. The product metric that matters is the former.
Design for caching. Stable system prompts, shared context prefixes and deterministic tool schemas all increase cache hit rates, and cached input is where the steepest discounts are.
Do not lock in. Multi-year committed-spend deals priced on today's rates are a bet against the clearest trend in the industry.

Prices will keep falling. Whether that helps your business depends less on the vendors than on whether your product is built to let it.

The EU AI Act's model rules, one year in: what changed for builders

GenAI Brief — Thu, 24 Sep 2026 00:00:00 GMT

The general-purpose AI (GPAI) provisions of the EU AI Act, Chapter V of the regulation, have applied since 2 August 2025. Enforcement powers for the Commission's AI Office follow a year later. That makes this a reasonable moment to set aside the speculation from 2024 and describe what the rules have actually meant in practice for the people building on and shipping these models.

Who is covered

The Act regulates providers of general-purpose AI models: the entity that develops a model and places it on the EU market, whether by API, by download or by embedding it in a product. It is separate from the risk-tiered rules for AI systems, which have their own timeline.

Three groups need to pay attention:

Labs releasing models, closed or open-weight. They are providers and carry the Article 53 obligations directly.
Companies that substantially modify a model. The Commission's guidelines say a downstream modifier becomes the provider of a new model when the modification is significant, with training compute relative to the original as the primary indicator. Light fine-tuning on a few thousand examples is unlikely to qualify; a large continued pre-training run is.
Deployers who build products on a model via API. They are generally not GPAI providers, but they inherit transparency duties under the AI system rules and depend on the upstream provider's documentation to meet them.

Figure 1: The first question is whether you are a provider, a modifier who has become a provider, or a deployer.

The baseline obligations

Every GPAI provider must:

Maintain technical documentation covering training, testing and evaluation, and make it available to the AI Office on request.
Provide documentation to downstream builders so they can understand the model's capabilities and limitations and meet their own obligations. In practice this is what a good model card has always been.
Have a copyright policy that respects EU copyright law, including honouring machine-readable opt-outs from text and data mining.
Publish a sufficiently detailed summary of training content, using the Commission's template.

Open-weight models released under a free licence are exempt from the first two items, but not from the copyright policy or the training-data summary. That exemption falls away entirely for models with systemic risk.

Systemic risk

A model is presumed to pose systemic risk when the cumulative compute used to train it exceeds 10^25 floating-point operations. The Commission can also designate models below that line. Providers of these models must additionally:

perform and document model evaluations, including adversarial testing;
assess and mitigate systemic risks across the model's lifecycle;
track and report serious incidents to the AI Office; and
ensure an adequate level of cybersecurity for the model and its infrastructure.

Most frontier releases from the major labs sit above the threshold. Most open-weight releases from smaller teams do not.

Figure 2: Every provider carries the middle column. Open-weight releases skip the documentation column unless the model carries systemic risk, which adds the third.

What actually changed

A year in, three practical effects stand out.

Model cards got longer and more uniform. The documentation requirements map closely onto what a thorough model card already contained, so the main change has been consistency: training-data summaries and copyright statements now appear where they were previously absent. See our guide on how to read a model card for what to look for.

The Code of Practice became the default compliance path. The Code is voluntary, but signatories get a presumption of conformity, and the AI Office has said it will focus enforcement attention elsewhere. Most large providers signed. A few declined publicly, citing legal uncertainty, and continue to argue their compliance case directly.

Downstream teams started asking for paperwork. Procurement questionnaires now routinely ask whether a vendor's models are GPAI-compliant and whether a Code signatory sits upstream. If you sell into European enterprises, expect the question.

What still applies to you

If you only call an API, your obligations under Chapter V are indirect. Your direct obligations arrive through the AI-system rules: labelling AI-generated content, informing users they are interacting with an AI, and, for high-risk use cases, the full conformity regime that phases in through 2027.

If you fine-tune, keep a record of the compute you used. It is the number a regulator will ask for first.

How to read a model card without getting fooled

GenAI Brief — Wed, 23 Sep 2026 00:00:00 GMT

Every model launch now comes with a card: a document that is part technical specification, part safety disclosure and part marketing. Knowing which part you are reading at any moment is the whole skill.

This guide goes through a typical card top to bottom and flags what to trust, what to verify and what to ignore.

The benchmark table

The centrepiece of most cards is a table comparing the new model to its competitors on a dozen benchmarks. Treat it as a claim, not a measurement, and read the footnotes before the numbers.

Things that legitimately move scores by several points and are routinely applied unevenly:

Sampling settings. Greedy decoding versus temperature sampling, and how many samples are drawn. "pass@1" with majority voting over 64 samples is not the same as one attempt.
Prompting. Zero-shot versus few-shot, and whether a chain-of-thought instruction was used. Cards frequently report their own model with the favourable setting and competitors with whatever number was published.
Tool access. Coding benchmarks in particular vary enormously depending on whether the model can run code, browse or retry.
Reasoning budget. For models that scale test-time compute, the score depends on how much thinking was allowed. A "high" setting can cost 20x the tokens of a "low" one.

A good card states all four for every row. A great card runs competitors under the same settings. If neither is true, the table tells you about the vendor's marketing team, not the model.

For the "why" behind this, read our editorial on why benchmarks have become a marketing channel.

Figure 1: Four settings have to match before two rows in a table can be compared. If they do not, the row is marketing.

Context window

The headline context length is a hard limit, not a capability. What you want to know is how much of that window the model can actually use. Look for:

a needle-in-a-haystack result across positions and depths, ideally with multiple needles;
a long-context reasoning benchmark rather than pure retrieval; and
any statement about quality degradation beyond a certain length.

If the card only lists the maximum, assume usable context is materially smaller and test it yourself before designing around it.

Training data and cutoff

Two questions matter for builders: when does the model's knowledge stop, and what kinds of data was it trained on? The knowledge cutoff should be a specific month. "Training data" sections have become more detailed under EU AI Act documentation requirements, but they still rarely tell you whether a benchmark's test set was in the training mix. Contamination is the elephant in every benchmark table.

Safety and evaluations

The safety section usually contains the most careful measurement in the whole document, because it is the part regulators read. It is worth reading for a different reason: the refusal behaviour it describes is what your users will hit. Over-refusal rates, jailbreak resistance and the categories of content the model declines all shape product behaviour directly.

Limitations

This is the section to read first if you read only one. A card that says "the model may hallucinate" and nothing else is not disclosing anything. A card that says "accuracy on numerical reasoning over tables drops sharply beyond roughly 40 rows" is giving you an engineering constraint. The specificity of the limitations section is the best single proxy for how honest the rest of the card is.

What is not in the card

Three things decide whether a model is viable for you and none of them lives in the card:

Price, including cached input, batch discounts and long-context surcharges. See who benefits from falling inference prices.
Rate limits and availability, especially at launch, when the best models are often effectively unavailable at production volume.
Data terms: retention, training on your inputs, regional hosting.

Read the pricing page and the terms of service alongside the card. Together they are the specification. Alone, the card is a brochure with citations.

A checklist

Before adopting a model on the strength of its card, you should be able to answer:

Under what settings were the benchmark numbers produced, and were competitors run the same way?
What is the usable context length, as opposed to the maximum?
What is the knowledge cutoff, to the month?
What specific failure modes does the vendor admit to?
What does it cost at your expected token mix, and can you get the rate limit you need?

If the card and its companion pages answer all five, you have enough to run your own evaluation. If they do not, that is the finding.

Open-weight models are closing the gap. The economics say it will not fully close

GenAI Brief — Tue, 22 Sep 2026 00:00:00 GMT

Three years ago the best open-weight model trailed the frontier by a year or more on nearly every benchmark. Today, on the standard suites, the lag is measured in months, and on price-performance the open models lead. That progress is real and it has changed what most teams should default to.

It has also produced a recurring prediction: that the gap will close entirely and closed models will become a premium brand for a commodity product. We think the prediction misreads where the gap comes from. It is not primarily technical. It is a question of who pays for frontier training runs and what they expect in return.

Where the gap is today

On knowledge, reasoning and coding benchmarks, leading open-weight releases from Chinese labs, from Meta, Mistral and a lengthening list of others have repeatedly matched closed models that were state of the art six to twelve months earlier. The distance is shortest on tasks with abundant public training data and clear evaluation signals. It is longest on:

long-horizon agentic tasks, where post-training on proprietary interaction data matters;
multimodal understanding at high resolution and across video; and
reliability at the tail: the rare, expensive failures that enterprise buyers care about most.

Those three categories are exactly where the closed labs are concentrating their effort, which is not a coincidence.

Why the gap exists

A frontier training run costs hundreds of millions of dollars in compute alone, before the research staff and the data. Somebody has to expect a return. For the closed labs the return comes from exclusive access: API revenue, consumer subscriptions and enterprise contracts that only exist because nobody else can serve the same model. Releasing the weights destroys that return.

Open-weight releases therefore come from organisations whose return does not depend on exclusivity:

Funder	Why release weights	What they tend to release
Hardware vendors	Every open model sells more chips	Models optimised for their own silicon
Cloud platforms	Hosting is the product; the model is a loss leader	Broad general-purpose models
Labs with a services business	Consulting, fine-tuning, on-prem deployments	Strong base models with permissive licences
National or strategic programmes	Sovereignty, talent, influence	Multilingual models and large releases with unusual timing
Labs seeking distribution	Mindshare that converts to a later closed tier	Excellent small and mid-size models

Figure 1: Open weights come from organisations whose return does not depend on exclusive access.

Each of these funders has a reason to stop at "good enough to be useful and widely adopted". None of them has a reason to spend a further billion dollars to be first by three months, because being first by three months is only worth a billion dollars if you can charge for exclusive access. That is the structural reason the frontier stays closed.

Figure 2: The gap at the frontier is a funding question, not a technical one.

Where that leaves builders

For most production workloads the decision is no longer binary. The practical questions are:

Is the task within reach of the current open frontier? For classification, extraction, summarisation, RAG and most coding assistance, yes, and has been for a while. Re-benchmark quarterly and you will keep finding that the answer expanded.
Where will you serve it? Self-hosting makes sense at high, steady volume or under data-residency constraints. Otherwise a hosted open-model provider is usually cheaper than owning GPUs.
What does the licence actually allow? "Open" spans everything from Apache 2.0 to licences with user thresholds, field-of-use restrictions and attribution requirements. Read it.
Who maintains the model? Open releases are snapshots. Security patches, tokenizer fixes and safety updates depend on the releasing lab continuing to care.

The frontier will stay closed for the tasks that need it. The rest of the industry is increasingly running on weights anyone can download, and that is the more consequential trend.

What "reasoning" models actually do differently

GenAI Brief — Sat, 19 Sep 2026 00:00:00 GMT

Since late 2024 every major lab has shipped a "reasoning" model, and the term has taken on a slightly mystical tone in product copy. The mechanism is less mysterious than the marketing, and understanding it is the difference between paying for reasoning where it helps and paying for it everywhere.

The mechanism

A reasoning model is an ordinary large language model that has been post-trained to generate a long sequence of intermediate tokens, a chain of thought, before it commits to an answer. The core observation, documented by Wei and colleagues in 2022, is that prompting a model to "think step by step" improves accuracy on multi-step problems. Reasoning models bake that behaviour in and, crucially, train it.

The training recipe that made the current generation possible is reinforcement learning on verifiable tasks. You take problems with checkable answers, such as maths competition questions, unit-tested coding tasks and logic puzzles, let the model generate long attempts, and reward the attempts that reach the right answer. The model learns, without step-by-step supervision, to plan, to check its own work, to backtrack when a line of attack fails and to spend more tokens on harder problems. DeepSeek's R1 report was notable for showing how much of this emerges from a fairly simple reward.

The result is a model that converts extra inference compute into extra accuracy. That is the property Snell and colleagues formalised as test-time compute scaling, and it is why "how long did it think" is now a dial you can turn.

Figure 1: Reinforcement learning on checkable problems teaches the model to spend more tokens when a problem is hard.

Where it helps

The gains concentrate on problems with two properties: the answer can be checked, and reaching it takes many dependent steps.

Mathematics and formal logic. The canonical case; the improvements are dramatic.
Code generation and debugging, especially when the model can run tests and iterate.
Structured planning: decomposing a task, sequencing tool calls, satisfying constraints.
Careful reading of long inputs where the answer depends on reconciling several passages.

Where it does not

Recall. Knowing a fact does not benefit from thinking longer. Reasoning models are not more knowledgeable than their base models.
Style, taste and tone. Writing tasks often get worse, or at least more laboured.
Latency-sensitive interactions. A chatbot that pauses for twenty seconds is a worse chatbot.
Simple classification and extraction at volume, where the extra tokens are pure cost.

Vendors increasingly ship a "thinking budget" or effort setting because of this. The same model with a low budget behaves like a fast conventional model; with a high budget it becomes a slow expert. Measure each production task at two or three budgets before choosing.

Figure 2: The dividing line is whether the answer can be checked and whether reaching it takes several dependent steps.

What the chain of thought is not

It is tempting to read the visible reasoning as a transcript of what the model "really did". Interpretability research has repeatedly shown that chains of thought can be unfaithful: the model may reach an answer for reasons that do not appear in the text, and the text may contain steps that did not influence the outcome. Some vendors now show summarised or hidden reasoning for this reason among others.

The practical rule: use the chain of thought for debugging, prompt design and building intuition. Do not build product logic that parses it, and do not present it to users as an explanation of the answer.

Deciding when to pay

A rough decision procedure that has held up well in practice:

Does the task have a checkable answer, or at least a clear success criterion? If not, reasoning is unlikely to help much.
Does solving it require holding several intermediate results? If it is a one-step lookup, no.
Can the user tolerate the latency? If not, consider running the reasoning model offline to generate training data or few-shot examples for a fast model instead.
Have you measured the accuracy gain against the cost increase on your own data? If not, you are guessing.

Reasoning models are a genuine capability step, arguably the most important since instruction tuning. They are also the easiest way to quadruple an inference bill for no benefit. The mechanism tells you which one you are getting. For the roadmap implications, see how test-time compute changed the scaling picture.

Test-time compute changed the scaling roadmap. Here is what it costs

GenAI Brief — Thu, 17 Sep 2026 00:00:00 GMT

From 2020 to 2024 the industry ran on a single story: bigger training runs produce better models, predictably, along a power law. Kaplan's scaling laws and the Chinchilla revision from Hoffmann and colleagues turned that story into a planning tool. You could estimate how much better the next model would be from how much compute you were prepared to buy.

The reasoning models that arrived from late 2024 introduced a second axis. If a model is trained to think longer, then for a fixed set of weights you can buy more accuracy at inference time, per query, by letting it generate more tokens. Snell and colleagues showed that on many tasks this is a better use of a marginal dollar than more pre-training. The roadmap did not break; it acquired a dimension. But the economics changed in ways that are still working through the industry.

From capex to opex

Under pure training scaling, the cost of intelligence was paid once, up front, by the lab. Serving the model was cheap and roughly constant per token. The customer paid for volume, not for quality.

Under test-time scaling, quality is bought per query. A hard question costs more to answer well than an easy one, and the customer sees that in their bill. Three consequences follow:

Model tiers are becoming effort tiers. Vendors expose thinking budgets, "effort" parameters or fast/slow variants of the same weights. Choosing a model now includes choosing how much to spend per call.
Margins depend on routing. Sending every query to the maximum effort setting is ruinous. The teams doing well route by difficulty, and difficulty estimation has become an engineering discipline of its own.
The lab's capital expenditure is no longer the only thing to track. Inference fleet size and utilisation matter as much as training clusters, and the labs are building both.

Figure 1: Both axes buy capability. Only one of them shows up on the customer’s invoice.

Benchmarks without a cost axis are misleading

A single accuracy number for a reasoning model is close to meaningless without the compute that produced it. The same model can score anywhere on a wide range depending on the budget, and the budget used for the marketing number is usually the maximum.

Responsible reporting now shows accuracy as a curve against tokens or dollars per problem. When comparing models, compare the curves, or at least compare points at matched cost. Our guide to reading a model card covers what to look for, and our editorial on benchmarks as a marketing channel covers why you will often not find it.

Pre-training did not stop

It is fashionable to say that pre-training scaling has "hit a wall". The evidence is weaker than the slogan. What has happened is that the cheapest capability gains moved: for a period, reinforcement learning on reasoning and better post-training delivered more per dollar than the next order of magnitude of pre-training. Labs with the money continue to do both, and the largest training runs in history are underway. The models that lead over the next two years are likely to be the ones that combine a bigger base with more effective test-time scaling, not those that pick one.

What this means if you build on these models

Measure accuracy per dollar, not accuracy. Run your evaluation set at several effort levels and plot it. The right operating point is rarely the maximum.
Route. A cheap classifier that decides which queries need deep reasoning pays for itself quickly.
Cache and distil. Expensive reasoning on a recurring question should happen once. Use the reasoning model to produce answers or training data, then serve a fast model.
Budget for variance. Per-query cost now depends on the question. Set limits.

Figure 2: Routing by query type is where most of the savings from test-time compute are captured.

The scaling story is still true. It is just that "scale" now has two meanings, and one of them shows up on your invoice.

Five LLM gateways compared: routing, failover, governance and where each fits

GenAI Brief — Tue, 15 Sep 2026 00:00:00 GMT

Once a company is calling more than one model provider from more than one application, the same code appears everywhere: credential handling, retries, fallbacks, spend tracking, logging. An LLM gateway collapses that into one service. The category has matured quickly, and the products now differ less on the basic feature list than on architecture, deployment model and operating philosophy.

This comparison looks at five gateways that represent the main approaches: two self-hosted open-source gateways, one hosted control plane, one extension of an existing API gateway and one edge-network offering. Details reflect public documentation at the time of writing; verify current specifics before deciding.

Figure 1: The value of a gateway is that this path is configured once instead of reimplemented in every application.

The contenders

Gateway	Model	Written in	Best fit
Bifrost	Self-hosted, open source (also hosted)	Go	High-throughput and latency-sensitive workloads; teams wanting a single binary
LiteLLM	Self-hosted, open source (also hosted)	Python	Broadest provider coverage; Python-native teams
Portkey	Hosted control plane (self-host option)	Managed	Teams that want observability and guardrails without running infrastructure
Kong AI Gateway	Plugin on Kong Gateway	Lua/Go on Kong	Organisations already standardised on Kong for APIs
Cloudflare AI Gateway	Hosted, edge	Managed	Teams on Cloudflare wanting caching, analytics and rate limits with minimal setup

What actually differs

Overhead at load

Every gateway adds latency. The question is how much, and how it behaves as requests per second climb. Go-based gateways such as Bifrost are designed around microsecond-scale added latency at thousands of requests per second on a single instance. Python-based gateways such as LiteLLM are easier to extend and have historically carried more overhead per request, which matters at high volume and less at low volume. Hosted gateways add a network hop whose cost depends on where your application and the provider are; edge-based gateways minimise it if your traffic is already on their network.

If you serve a chat product at low volume, none of this matters. If you serve an agent platform making millions of tool-calling requests a day, it dominates.

Failover semantics

All five retry and fall back. The differences are in what they distinguish:

Dead key versus transient error. A revoked credential should be rotated out immediately; a 5xx from the provider should be retried with backoff. Gateways that treat both the same either hammer a dead key or give up too early on a live one.
Provider-level versus model-level fallback. Falling back from one provider's model to an equivalent model at another provider requires a mapping the gateway has to maintain.
Load balancing across keys and regions. Useful for staying under per-key rate limits and for latency.

Bifrost and LiteLLM expose the most configuration here; Portkey's fallback and load-balancing "configs" are expressive and versioned; Kong and Cloudflare cover the common cases with less granularity.

Governance and policy

Governance means: who can call what, with what budget, and what gets logged. The models differ:

Virtual keys and budgets per team are table stakes and all five offer some form.
Guardrails (PII redaction, content filters, prompt-injection checks) are strongest in the hosted control planes and in gateways with plugin systems.
Policy as code. Kong's declarative configuration integrates with existing API governance; Bifrost and LiteLLM configure through files or a UI; Portkey and Cloudflare configure through their dashboards and APIs.

Deployment and data residency

This is the constraint that most often decides the choice on its own. A hosted-only gateway cannot serve an air-gapped or VPC-isolated deployment, and a hosted gateway means prompts and completions transit a third party. Self-hosted gateways keep everything inside your boundary at the cost of running them. Kong inherits whatever deployment model your Kong installation has.

Figure 2: One constraint usually settles the choice before any feature comparison matters.

Recommendations

You have data-residency or air-gap requirements: self-host. Choose Bifrost if throughput and latency matter and you want minimal operational surface; choose LiteLLM if your team is Python-first and you value its very wide provider list.
You want the least operations and strong observability: Portkey, or Cloudflare AI Gateway if you are already on Cloudflare and your needs are caching, analytics and limits rather than deep policy.
You already run Kong: Kong AI Gateway extends the platform you have, and consistency across your API estate usually outweighs feature-by-feature comparisons.

Whatever you pick, put an evaluation harness behind it. A gateway makes switching models trivial, and re-benchmarking quarterly is where the savings come from.

GenAI Brief has no commercial relationship with any of the vendors above. Feature claims are drawn from public documentation.

The state of AI agent frameworks in 2026: a survey

GenAI Brief — Sat, 12 Sep 2026 00:00:00 GMT

Two years ago "agent framework" meant a handful of Python libraries wrapping a while-loop around a model call. Today it is a crowded category with distinct schools of thought, backed by the labs themselves as well as startups and open-source communities. This survey maps the landscape as it stands, describes the design bet each camp is making and lists the questions that should decide your choice.

We deliberately avoid feature matrices. Features converge quickly; architectures do not.

Four camps

1. Graph orchestrators

Representative: LangGraph, and similar state-machine libraries.

The bet: an agent is a graph of nodes (model calls, tool executions, human checkpoints) connected by edges with conditions. State is explicit, persisted and inspectable. This makes complex topologies, cycles with exit conditions, parallel branches and approval gates tractable, and it makes durable execution (pause, resume, replay) natural.

The cost is ceremony. Simple agents become verbose, and the graph abstraction leaks into every part of the code. Teams with long-running, multi-step workflows and compliance requirements tend to land here.

2. Lab-native SDKs

Representative: the agent SDKs from OpenAI, Anthropic and Google.

The bet: the lab knows its own models best, and an SDK that exposes handoffs, tools, guardrails and tracing with a minimal surface will beat generic frameworks on reliability with that lab's models. These SDKs are typically small, opinionated and well integrated with the vendor's hosted tools (web search, code execution, file search) and observability.

The cost is portability. Most support other providers to some degree, but the idioms and the best-tested paths follow one vendor's models. For teams committed to a provider, they are often the fastest route to a working system.

3. Multi-agent role frameworks

Representative: CrewAI, AutoGen (now AG2), and Microsoft's Semantic Kernel agent features.

The bet: complex tasks decompose into roles, and agents with distinct personas, tools and goals collaborating through conversation produce better results than one agent with one long prompt. The mental model is a team, and the framework provides the meeting room.

The cost is unpredictability and tokens. Conversations between agents are expensive and hard to steer, and evaluation is difficult because failures are emergent. These frameworks shine for exploratory and creative workloads and struggle where determinism matters.

4. Typed minimalists

Representative: Pydantic AI, Mastra, the Vercel AI SDK.

The bet: the model call is the primitive, structured outputs and type safety are the main source of reliability, and everything else should be plain code in the host language. These libraries feel like ordinary application frameworks with a model client attached, and they integrate naturally with existing web stacks, especially in TypeScript.

The cost is that you build orchestration yourself. For many production agents, which have a small, fixed number of steps, that is a feature.

Figure 1: Each camp trades a different thing for its convenience.

Convergence at the protocol layer

The most important development of the last year is not a framework. It is that the boundaries between agents and their tools, and between agents and each other, are standardising. The Model Context Protocol has become the default way to expose tools and data to any agent, regardless of framework. Agent-to-agent protocols are earlier but moving in the same direction.

The effect is to make framework choice less consequential. A tool built once as an MCP server works from any camp. Switching frameworks used to mean rewriting integrations; increasingly it means rewriting only control flow.

The problems nobody has solved

Every camp struggles with the same three things:

Evaluation. Agents fail in long-tailed, path-dependent ways. Unit tests catch little; production traces catch everything too late. Trace-based evaluation with rubric-scoring models is the current best practice and is still immature.
Durable execution. Tasks that run for minutes or hours need to survive process restarts, provider outages and human delays. Graph orchestrators handle this best; the rest bolt it on.
Cost control. Agents spend tokens non-deterministically. Budgets per run, routing by difficulty and aggressive caching are necessary in every framework and native in none. Our analysis of test-time compute economics covers why this is getting harder, not easier.

Choosing

Ask, in order:

Is my agent's topology known and small? If yes, a typed minimalist library or a lab SDK is enough. If it is a large or evolving graph with human steps, a graph orchestrator.
Am I committed to one model provider? If yes, that provider's SDK is probably the most reliable path. If not, avoid anything whose best-tested path is single-vendor.
Do I need durable, resumable execution? If yes, this narrows the field sharply.
What does my team already write? TypeScript teams should look hard at the TypeScript-native options; Python teams have the most choice.
How will I evaluate it? If the answer is "we will look at the logs", stop and fix that first. It is framework-independent and it is the thing that will decide whether the agent ships.

Figure 2: Four questions narrow the field to one camp.

The frameworks will keep changing. The protocols and the evaluation discipline are the durable investments.

Benchmarks became a marketing channel. Treat them like one

GenAI Brief — Thu, 10 Sep 2026 00:00:00 GMT

There was a period, roughly 2019 to 2023, when a new benchmark could tell you something about the state of the field. Someone would publish a hard test, models would score badly, and the gap between the score and human performance was a reasonable proxy for how far the technology had to go.

That period is over. Benchmarks are still published, still cited and still improve, but their primary function has changed. A benchmark table is now the centrepiece of a launch post, produced by a marketing team from numbers selected by a research team, under settings chosen to produce the best-looking row. It is collateral. This is not a conspiracy and it does not require anyone to act in bad faith. It is what happens when a measurement becomes a target for organisations with billions of dollars at stake.

Why it was inevitable

Three forces made it so.

Contamination is structural. Frontier models are trained on very large fractions of the public internet. Any benchmark whose questions or close paraphrases are public will leak into training data eventually, and the labs' decontamination efforts, while real, are imperfect and unverifiable from outside. A score on a public benchmark is, in part, a measure of how much of that benchmark the model has seen.

Saturation is fast. Benchmarks that took years to approach human parity now saturate within months of release, because the labs are specifically optimising for the capabilities they test. Once a benchmark saturates, differences between models are noise, and noise is reported as leads.

Settings are a free variable. Number of attempts, sampling temperature, prompting strategy, tool access and, for reasoning models, thinking budget all move scores by several points. Every launch table picks the favourable combination for its own model. Our guide to model cards walks through the footnotes to check; our analysis of test-time compute explains why the thinking budget, in particular, can swing a result across a wide range.

Figure 1: The lifecycle of a benchmark ends in a launch table. Nobody has to act in bad faith for this to happen.

What the tables are still good for

None of this makes benchmark tables worthless. They are good for two things.

First, ruling models out. A model that scores poorly on a relevant benchmark under generous settings is unlikely to surprise you in production. The tables are a floor, not a ceiling.

Second, tracking the field's direction. Aggregated across many models and many months, the benchmarks describe which capabilities are improving fastest. That is useful for planning, even when individual comparisons are unreliable.

What the tables are not good for is the thing launch posts use them for: deciding that model A is better than model B for your task.

What builders should do

The answer is not to wait for a better benchmark. Any benchmark that becomes widely cited will be optimised for. The answer is evaluation that vendors cannot optimise for because they cannot see it:

Build a private evaluation set from your own production traffic and your own definition of success. Fifty carefully labelled examples beat a thousand from a public dataset.
Report accuracy against cost. A point estimate hides the thinking budget. A curve does not.
Re-run it on every candidate model under the same settings. This is the only comparison that is actually fair.
Treat vendor numbers as a screening tool, and say so internally, so nobody mistakes the brochure for the measurement.

Figure 2: A private evaluation set is the only measurement a vendor cannot optimise against.

What the industry should do

Independent, cost-adjusted, contamination-resistant evaluation is a public good, and public goods are underfunded. The organisations that run held-out leaderboards, private test sets and live human-preference arenas are doing the field's most important measurement work on a fraction of the budget of a single training run. Labs that want their benchmark claims believed should fund them, and should publish their numbers under the independent bodies' settings rather than their own.

Until then, read the footnotes, run your own tests and remember what a launch table is for. It is for launching.

The real cost of running an AI product, line by line

GenAI Brief — Tue, 08 Sep 2026 00:00:00 GMT

Ask a team what their AI feature costs and they will quote a token bill. It is the most visible number, it arrives monthly from the provider and it is the one the finance team asks about. It is also, in most production systems we have looked at, well under two-thirds of the true cost, and frequently under half.

This is a line-by-line breakdown of where the rest goes, with the metrics that make each line manageable.

1. Inference

The token bill: input tokens, output tokens, and increasingly separate prices for cached input, long context and reasoning tokens. It is the line that falling model prices improve, and the only one.

Manage it with: cost per completed task, not cost per token; a cache hit rate target; and a per-task token budget with a hard cap. For agentic features, track the distribution of cost per task, because the mean is set by a small number of runaway runs.

2. Retrieval and context infrastructure

Anything that assembles the prompt: vector databases, search indexes, embedding generation, document processing pipelines and the storage behind them. These scale with corpus size and query volume, and embedding costs recur every time you change the embedding model or the chunking strategy.

Manage it with: cost per query for retrieval separated from inference, and a re-indexing budget line for model changes.

3. Evaluation

Every prompt change, model swap and retrieval tweak needs to be checked against a test set before it ships. Doing that properly means labelled data, a harness, model-graded rubrics (which themselves cost tokens) and periodic human review of the grader.

Teams that skip this line pay for it in the next one instead. Our survey of agent frameworks found evaluation to be the least-solved problem in every camp, and our editorial on benchmarks as marketing explains why public numbers cannot substitute for it.

Manage it with: evaluation cost as a percentage of inference spend (a healthy figure is often 10 to 20 percent), and a fixed evaluation budget per model swap.

4. Observability and logging

Storing prompts, completions, tool calls and traces at production volume is a real storage and processing cost, and the tooling to search and analyse them is either a vendor bill or engineering time. Retention policies matter: full-fidelity traces for thirty days and sampled traces after is a common compromise.

Manage it with: sampling rates tied to feature maturity, and retention tiers.

5. Human review and escalation

Any feature with a quality bar has humans behind it: reviewers checking a sample of outputs, support staff handling escalations when the model is wrong, and domain experts labelling data for evaluation. This line is invisible in the cloud bill and often the largest single cost in regulated domains.

Manage it with: review rate as a first-class metric with a plan to reduce it as measured quality rises.

6. Engineering time on change

Models are deprecated, prices move, better options appear. Each swap costs engineering time to re-prompt, re-evaluate and re-tune, plus the evaluation spend from line 3. Teams that swap models quarterly to capture price cuts need to budget the swap itself.

Manage it with: a standing "model change" budget rather than treating each as a project.

7. Guardrails, gateways and safety

Content filters, PII redaction, prompt-injection checks and the gateway that routes and logs everything. Often a per-request cost on top of inference, and often an extra model call.

Manage it with: per-request overhead tracked separately from inference so its cost is visible.

Figure 1: Costs that scale with usage and costs that scale with change need different controls.

Putting it together

A representative breakdown for a mid-scale production feature, from our conversations with teams, looks roughly like this:

Line	Share of total	Scales with
Inference	30–60%	Usage
Retrieval infrastructure	5–15%	Corpus and usage
Evaluation	5–15%	Rate of change
Observability	5–10%	Usage
Human review	5–30%	Quality gap
Engineering on change	5–15%	Rate of change
Guardrails and gateway	2–8%	Usage

The ranges are wide because the mix varies enormously by product. The point is not the numbers; it is that a cost model with one line cannot tell you which optimisation to make. Build all seven, and the answer to "why did the bill go up" becomes a lookup rather than an investigation.