Benchmarks became a marketing channel. Read the footnotes, then run your own

A benchmark table on a model launch page is not a measurement. It is a claim about a measurement, assembled by the people whose model it is, under conditions they chose, against competitor numbers they usually did not re-run. That is the state of the industry in September 2026, and we think it is time to read launch tables the way we read any other collateral: for what is in the footnotes, not the headline row.

This is not an accusation of fraud. The people running evaluations at the frontier labs are, by every account we have read, careful. The problem is structural. When a number decides press coverage, enterprise procurement and, increasingly, the valuation of the company reporting it, the number becomes a target, and a target stops being a measurement. The evidence for that has become overwhelming over the past two years, and it comes from the labs themselves as much as from their critics.

What follows is the case, with numbers. Then the counter-argument, which is stronger than critics admit. Then a concrete protocol for reading a launch table and for building the only evaluation a vendor cannot optimise against: your own.

From research artefact to launch-day table

Every important benchmark has followed the same arc. A research group publishes a hard test. Models score badly. The gap to human performance is a useful proxy for how far the technology has to go. Then the labs start optimising for it, the gap closes, the benchmark saturates or leaks, and the number that once meant something becomes the centrepiece of a launch post. The interval between those two states has shrunk from years to months.

GPQA is the clean example. When it was published in November 2023, the paper reported that PhD-level experts scored 65% on its 448 main-set questions (74% after discounting clear mistakes), that skilled non-experts with more than thirty minutes of unrestricted web access scored 34%, and that the strongest GPT-4 system scored 39%. The Diamond subset, 198 questions on which both expert validators were right and most non-experts were wrong, put adjusted expert accuracy at 81.3%. Two years later Google’s Gemini 3 launch post reported 91.9% on GPQA Diamond without tools, and OpenAI’s GPT-5.2 post reported 93.2% for GPT-5.2 Pro. A test designed to be Google-proof for experts is now a rounding error between frontier models.

SWE-bench Verified is the more instructive case, because it was created to fix a benchmark and then had to be retired itself. In August 2024 OpenAI announced that 93 Python developers had annotated 1,699 SWE-bench tasks and found 38.3% had underspecified problem statements and 61.1% had unit tests that could unfairly fail valid solutions; 68.3% of the original set was filtered out, leaving 500 verified tasks. GPT-4o went from 16% on the original set to 33.2% on the verified one. By late 2025 every frontier vendor was reporting between 76% and 81% on it. In February 2026 OpenAI published a post explaining why it would no longer report the benchmark. Its audit of 138 problems that o3 did not consistently solve across 64 runs found that 59.4% had material issues in test design or problem description: 35.5% had tests that enforced specific implementation details, 18.8% had tests checking functionality the problem never specified, and 5.1% had miscellaneous problems. More damning, GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash were each shown reproducing exact gold patches, class and method names, verbatim inline comments and line numbers from the original pull requests. OpenAI now recommends SWE-bench Pro and, in its words, recommends other developers do so too.

Humanity’s Last Exam ran the arc fastest. The paper, submitted in January 2025 with more than a thousand contributing authors, motivated itself by noting that models were already above 90% on MMLU. Its 2,500 questions were built to resist retrieval and to be unambiguous to grade. Within ten months the lastexam.ai leaderboard showed Gemini 3 Pro at 38.3%, and vendors were reporting tool-assisted scores above 50%. In September 2026 the Center for AI Safety and Scale AI released HLE-Diamond, a cleaned 1,000-question subset, after what they describe as a year-long refinement process; its top reported score without tools was 60.6%.

ARC-AGI-1, first released in 2019, was declared effectively done in December 2024 when ARC Prize verified OpenAI’s o3 at 75.7% on the semi-private set, and its successor ARC-AGI-2 launched three months later with pure LLMs at 0% and the best public reasoning system at 4%. That story has one difference from the others which we return to below: ARC Prize insisted on cost per task from the start.

Figure 1: The lifecycle of a benchmark ends in a launch table. Nobody has to act in bad faith for this to happen.

The mechanics of inflation

Five mechanisms turn a measurement into a marketing number. They compound.

Contamination is the default, not the exception

Frontier models are trained on very large fractions of the public internet, and every public benchmark is on the public internet. The scale of ordinary leakage was documented as early as February 2024, when Balloccu and colleagues reviewed 255 papers that evaluated GPT-3.5 and GPT-4 and estimated that roughly 4.7 million samples from 263 benchmarks had been sent to those models through the API in the year after release, where OpenAI’s terms at the time allowed them to be used for training. The authors called this indirect leaking: no lab needs to scrape a benchmark when the research community submits it one prompt at a time.

For code the evidence is more specific. SWE-Bench+, from October 2024, found that 32.67% of successful SWE-bench patches by one agent were cases where the solution was already present in the issue text or comments, that a further 31.08% passed only because the tests were too weak, and that a filtered resolution rate for SWE-Agent with GPT-4 fell from 12.47% to 3.97%. Over 94% of the issues predated the models’ training cutoffs. The SWE-Bench Illusion, from June 2025, showed that frontier models could identify the buggy file path from the issue description alone up to 76% of the time on SWE-bench Verified repositories but only up to 53% on repositories outside the benchmark, and could reproduce functions with up to 35% consecutive five-gram overlap on SWE-bench against 18% elsewhere. Neither task is solvable without either seeing the repository or remembering it.

The labs know this. Anthropic’s Claude Opus 4.7 announcement in April 2026 carries the footnote: “Our memorization screens flag a subset of problems in these SWE-bench evals. Excluding any problems that show signs of memorization, Opus 4.7’s margin of improvement over Opus 4.6 holds.” That is good practice and we welcome it. It is also an admission that the headline number includes problems the vendor’s own screens think the model may have memorised. And leakage is not only in the weights. The SWE-Bench Pro Verified paper from September 2026 documents reward hacking enabled by leaked gold solutions and hidden evaluation information inside the benchmark environment itself, and finds that after closing those channels “some models perform substantially worse than previously evaluated”. Epoch AI’s own SWE-bench Verified harness removes git history after each issue’s date precisely because an agent that can run git log does not need to have memorised anything.

Saturation turns differences into noise

A benchmark on which the field’s best models score 92% has about 16 questions of headroom out of 198. At that point a two-point lead is a handful of questions, and a handful of questions is within run-to-run variance for a sampled model. Epoch AI reports confidence intervals of plus or minus one standard error on its GPQA Diamond runs, which it repeats 16 times per model; vendors, with few exceptions, report a point estimate. Epoch built its Capabilities Index using item response theory, anchored with Claude 3.5 Sonnet at 130 and GPT-5 at 150, for exactly this reason: individual benchmarks saturate too quickly to track progress, so it fits a latent capability across many of them. NIST’s CAISI uses the same statistical family, treating models as students and benchmark items as exam questions.

Evaluation conditions are a free variable

This is where the launch tables stop being comparable, and it is best shown rather than argued. Below is one benchmark, SWE-bench Verified, as reported by five vendor documents between August 2025 and February 2026, with the conditions each document states.

Seven documents, at least five task subsets or scaffolds, attempt counts from one to twenty-five, thinking budgets from none to 200K tokens, and one number produced by a best-of-N selector. Every cell is honestly footnoted. None is comparable with the cell above it. Our guide to reading a model card walks through the same footnotes in more detail; our piece on test-time compute explains why the thinking budget in particular can move a reasoning model’s score across a wide range.

Tool access is the second free variable, and it matters most on the benchmarks that are supposed to be hardest. Google’s Gemini 3 Pro documentation reports 37.5% on Humanity’s Last Exam without tools and 45.8% with search and code execution, run with a domain blocklist so the model cannot retrieve benchmark answers. OpenAI’s GPT-5.2 post reports 34.5% without tools and 45.5% with search and Python for the same model, and 50.0% with tools for GPT-5.2 Pro. Anthropic’s Opus 4.6 post reports 53.0% with tools, where “with tools” means web search, web fetch, code execution, programmatic tool calling, context compaction and up to 3 million total tokens at maximum effort. xAI’s Grok 4 post reported Grok 4 Heavy as “the first to score 50.7% on Humanity’s Last Exam (text-only subset)”, with Python and internet tools. Four vendors, four definitions of the same benchmark, each headlined as a record.

The third free variable is who ran the competitor’s number. Google DeepMind’s methodology note is unusually candid: “All the results for non-Gemini models are sourced from providers’ self reported numbers unless mentioned otherwise”, with Humanity’s Last Exam figures for competitors taken from Scale’s leaderboard and Artificial Analysis “due to result availability”. Anthropic went the other way for one agentic benchmark in its Opus 4.5 post, re-running Gemini 3 and GPT-5.1 on an improved hosting environment and reporting their scores rising to 56.7% and 48.6% “from the values reported by their developers”. Both are defensible choices. Neither produces a table where every row was measured the same way.

Private variants and the leaderboard that was not a leaderboard

Human-preference arenas were supposed to be immune to contamination, because the test is live. They turned out to be vulnerable to something else. On 5 April 2025 Meta’s Llama 4 launch post stated that “Llama 4 Maverick offers a best-in-class performance to cost ratio with an experimental chat version scoring ELO of 1417 on LMArena.” Three days later LMArena posted that “Meta’s interpretation of our policy did not match what we expect from model providers. Meta should have made it clearer that ‘Llama-4-Maverick-03-26-Experimental’ was a customized model to optimize for human preference.” The model on the leaderboard was not the model in the download.

Three weeks later, The Leaderboard Illusion, from researchers at Cohere Labs, Princeton, Stanford, the Allen Institute, Waterloo and Washington, quantified the practice from about 2 million battles across 243 models and 42 providers between January 2024 and April 2025. Meta had tested 27 private variants before the Llama 4 release and Google 10; academic groups tested none. Of the 243 public models, 205 had been silently deprecated, against 47 officially listed, and 64% of the silently removed were open-weight or open-source. In a controlled experiment, training on Arena data lifted a model’s win rate on ArenaHard from 23.5% to 49.9%, a 112% relative gain. The authors recommended prohibiting score retraction, capping concurrent private variants at three, and publishing every tested model.

Arena’s response disputed the framing. Its pre-release testing policy had been public since 1 March 2024, it said; “any model provider can submit as many public and private variants as they would like, as long as we have capacity for it”; open models were 40.9% of the leaderboard once Llama and Gemma were counted. It also committed to marking scores provisional until 2,000 post-release votes accrue when a provider has pre-tested more than ten models. We think both sides are right about the facts and the paper is right about the incentive. If a lab can test 27 variants and publish one, the leaderboard reports the maximum of 27 noisy draws, not the expected value of the model you can buy.

When the funder is the examinee

The last mechanism is the quietest. In January 2025 Epoch AI clarified that OpenAI had commissioned the 300 core FrontierMath problems, owned them, and had access to the problems and solutions except for a holdout set, which was still being finalised at the time o3’s FrontierMath result was announced. “Our agreement did not prevent us from disclosing to our contributors that this work was sponsored by an AI company”, Epoch wrote, and the contributing mathematicians had not been told. Epoch has since managed the arrangement well: its March 2026 report on GPT-5.4 lists exactly what OpenAI can see (all 290 Tier 1 to 3 problems, solutions to 237 of them, 28 of 48 Tier 4 problems) and reports the holdout results separately: 42% on held-out Tier 1 to 3 against 50% overall, 55% on held-out Tier 4 against 38%, differences it calls not statistically significant. That is the right disclosure. It is also a reminder that the hardest benchmark in mathematics was paid for by the lab most interested in its result, and that nobody outside Epoch and OpenAI can check the questions.

What the independent evaluators find

A small number of organisations now run models themselves, under their own conditions, on data the vendors cannot see. Their results are the closest thing the field has to measurement.

Three findings from this group stand out.

The gap between vendor tables and held-out tests is large. CAISI’s May 2026 evaluation of DeepSeek V4 Pro noted that the vendor’s self-reported benchmarks put it level with frontier US models, then measured it on nine benchmarks across five domains, several of them non-public. On the ARC-AGI-2 semi-private set it scored 46% against 79% for GPT-5.5; its IRT-estimated capability was 800 (plus or minus 28) against 1,260 for GPT-5.5, which CAISI translates as roughly eight months behind the frontier. Scale’s SWE-Bench Pro, built from copyleft repositories and proprietary startup codebases specifically to resist contamination, put Claude Opus 4.1 at 17.8% and GPT-5 at 15.7% on its 276-task commercial set in the same season those models were reported above 70% on SWE-bench Verified. MLCommons’ August 2026 guide to trusting benchmarks cites SWE-bench evidence of 55-point gaps between public and private test sets as its example of why test-set leakage is a “structural, recurring threat”.

Cost changes the ranking. ARC Prize’s o3 verification in December 2024 reported 75.7% on the semi-private set at 6 samples per task, 33.5 million tokens and roughly 26 dollars a task at retail pricing, which qualified for the public leaderboard, and 87.5% at 1,024 samples, 5.7 billion tokens and 172 times the compute, which did not. Its December 2025 results list Gemini 3 Pro at 31% for 0.81 dollars a task, Claude Opus 4.5 with 64K thinking at 37.6% for 2.20 dollars, and Gemini 3 Pro wrapped in Poetiq’s refinement harness at 54% for 30 dollars; the winning Kaggle entry, restricted to about 50 dollars of compute, reached 24.03% on the private set at 0.20 dollars a task. ARC Prize reports that an average human scores 60% on ARC-AGI-2 and that every task has been solved by at least two people in under two attempts, at about 17 dollars a task. Which of those systems is “best” depends entirely on which axis you read. A single-column launch table cannot show that.

Independent re-runs mostly agree with vendors on the easy part. Epoch notes that GPT-5 Mini’s official 59.9% on SWE-bench Verified would be 59.8% on Epoch’s 484-task subset, so subset choice alone is not the problem. The problem is what the score means: on a contaminated benchmark, a faithfully reproduced 80% is still a faithfully reproduced measure of something other than coding.

The case for the tables, taken seriously

We would be making the same mistake as the launch posts if we left it there. The counter-argument has three parts, and each is partly right.

First, public benchmarks are still the best free signal. GPQA did track real progress from 39% to above 90%, and a model that scores 40% on it today is not a frontier model, whatever its marketing says. Time-horizon data from METR, aggregated across many models and years, describes a trend that no single launch table could. The tables are useful for ruling models out and for reading the direction of the field. What they are not useful for is the thing launch posts use them for: ranking two frontier models for your task.

Second, the labs have done real work on quality. SWE-bench Verified was itself an expensive cleanup that removed two thirds of a flawed dataset. Google’s blocklist on tool-assisted runs, Anthropic’s memorisation screens, OpenAI’s 64-run audit, Anthropic’s 25-trial averages and Google’s explicit statement of where every competitor number came from are all practices that did not exist in 2023. The footnotes have got better. Our complaint is that the headlines have not followed them.

Third, independent evaluation is expensive and someone has to pay for it. ARC Prize caps verified runs at 10,000 dollars and still needs zero-retention agreements with the labs. Epoch’s benchmarking is funded by a government institute and its hardest benchmark by a lab. CAISI is a government office. METR depends on pre-release access. Every one of these arrangements involves a compromise a purist would reject, and every one of them produces better evidence than the alternative of no independent number at all. The right response to a conflict of interest that is disclosed and managed, as Epoch’s now is, is to read the disclosure, not to discard the data.

Where we part company with the defence is on the claim that a well-footnoted vendor table is good enough for a buyer. The evaluation-condition table above shows why: a diligent reader can reconstruct what each vendor did, but cannot reconstruct a fair comparison, because the fair comparison was never run. That is not a footnote problem. It is a measurement problem that footnotes disclose and do not fix.

How to read a launch table

Governance is arriving. NIST’s draft AI 800-2, published in January 2026, breaks an evaluation protocol into inference settings (temperature, reasoning effort), scaffolding settings (tool availability, best-of-N aggregation), task settings and scoring settings, recommends tool-ablation experiments to gauge robustness, and asks evaluators to report uncertainty from both sampling variation and item selection, with pass@k named explicitly as an aggregation choice to be stated. MLCommons’ August 2026 guide poses seven questions, from relevance and data provenance to gaming prevention and a maintenance plan with retirement criteria. Neither is binding on a launch post. Until one is, the reader has to do the work. This is the protocol we use.

1. Find the conditions line before the numbers. Every table above has one; it is usually below the image, in smaller type. Extract five values: attempts (pass@1 or best-of-N), repeats (how many runs were averaged), tools (none, code, search, browsing), thinking budget or reasoning effort, and task subset (500, 477, 484, text-only, semi-private). If any is missing, treat the number as unverified.
2. Check who measured the competitors. If the note says self-reported, the competitor row is a different experiment. If the vendor re-ran competitors, look for the harness and whether the competitor’s own recommended settings were used.
3. Discount saturated benchmarks to zero. Anything where the frontier is above 90%, and anything the benchmark’s own maintainers have retired or re-issued (SWE-bench Verified, HLE without Diamond, ARC-AGI-1), tells you nothing about the difference between two frontier models.
4. Prefer verified rows from a third party. An “ARC Prize Verified” tag, an Epoch or CAISI run, or a Scale private-set score outranks a vendor’s own figure on the same benchmark. Check the date; verified scores lag launches by up to 30 days under ARC Prize’s policy.
5. Ask for the cost column. A tool-assisted, high-effort, best-of-N score that costs 30 dollars a task is a different product from a 2-dollar pass@1. If the post does not say what the headline run cost, assume it was not cheap.
6. Look for a contamination statement. Memorisation screens, canary strings, decontamination blocklists and held-out splits are now standard among the careful. Their absence in 2026 is a choice.
7. Read the footnote about “improvement over our previous model” separately from the competitor comparison. The first is usually measured under identical conditions and is the most reliable number in the post. The second usually is not.

Figure 2: A launch-table score survives as evidence only if it passes four questions. Most headline rows fail the second.

Building the evaluation a vendor cannot optimise for

Any benchmark that becomes widely cited will be optimised for. The only evaluation that escapes that is one the vendor cannot see: a private set built from your own traffic, scored on your own definition of success, run under settings you control. This is not a novel idea; it is what ARC Prize, Scale and CAISI do at scale, and what NIST’s draft describes for evaluators generally. The difference is that you can do it for one task and a few thousand dollars of inference.

Sample from production, not from a dataset. Pull real inputs, including the ugly ones, and include the distribution of lengths, languages and edge cases you actually see. The failure modes that matter to you are almost never the ones a public benchmark was built to test.

Write the criterion before you look at any model’s output. A pass or fail rule per item, reviewed by someone who owns the outcome, is worth more than a rubric a model grades. Where an LLM judge is unavoidable, MLCommons’ guidance applies: anchor it to human judgements on a documented sample and check it for bias and stability.

Fix every setting and run every candidate the same way. Same tools, same thinking budget, same attempts, same temperature, several repeats, confidence intervals. This is the comparison no launch table gives you, and it is the only one that is fair. Our explainer on what reasoning models actually do covers why a thinking budget is a setting to fix, not a feature to leave at default.

Report accuracy against cost, not accuracy. A point estimate hides the budget; a curve does not. ARC Prize’s scatter of score against cost per task is the right shape for an internal dashboard too.

Hold the set back and refresh it. Never paste your evaluation set into a public issue tracker, a shared prompt library or a vendor support ticket. Retire items that every model passes; add items from the failures you see in production. MLCommons’ “maintenance plan with retirement criteria” applies to a 200-item internal set exactly as it does to a public leaderboard.

Use the launch table as a screen, and say so. A model that scores poorly under a vendor’s own most generous conditions is unlikely to surprise you. Write that down as the purpose of vendor numbers in your evaluation policy, so that nobody in procurement mistakes the brochure for the measurement.

Figure 3: A private evaluation set is the only measurement a vendor cannot optimise against.

The closing argument

The industry’s answer to every benchmark that fails is a harder benchmark: Verified after SWE-bench, Pro after Verified, Diamond after HLE, ARC-AGI-2 after ARC-AGI-1, and now ARC-AGI-3 in the ARC Prize policy. Each is better than its predecessor. Each will follow the same arc, faster, because the incentive that bends the measurement has not changed and the labs have more capital to bend it with than ever.

We do not think the fix is to stop publishing tables. We think it is to stop treating them as findings. The labs should publish under a common, third-party protocol of the kind NIST has drafted, with the cost column and the uncertainty that Epoch, ARC Prize and CAISI already report, and they should fund the independent bodies that do the field’s most important measurement on a fraction of what any of them spends on a launch video. Vendors that want their numbers believed should want to be measured by someone else.

Until then, the responsibility sits with the reader. Read the footnotes. Discount the saturated rows. Trust the verified ones. Build your own set and run every model through it the same way. And remember what a launch table is for. It is for launching.