Benchmaxxing: How AI Labs Cherry-Pick Scores to Win Press Cycles

On December 20, 2024, OpenAI announced that its o3 model had solved more than 25% of FrontierMath, a benchmark on which prior frontier models had cleared roughly 2%. The funding behind that benchmark was disclosed only around the same launch window: OpenAI had paid for the dataset and held exclusive access to most of its hardest problems and their solutions, a relationship that earlier versions of the benchmark's paper had omitted entirely (Search Engine Journal). When Epoch AI ran its own independent test, released April 18, 2025, the public o3 scored closer to 10% (TechCrunch). Two numbers, the same model, a 2.5x gap that nobody outside the lab could have predicted on launch day.

That gap is not an accident, and it is not the story of one company having a bad week. It is the predictable output of a system that rewards a specific behavior, and that behavior now has a name.

What benchmaxxing actually means

Benchmaxxing is the practice of optimizing a model for what a leaderboard measures rather than for the capability the leaderboard is meant to proxy (jeannelizabeth.com). The distinction matters because every benchmark is a stand-in. A coding benchmark is not "the ability to write software"; it is a finite set of tasks that correlate, imperfectly, with that ability. The moment a vendor begins improving the proxy without improving the underlying skill, the correlation breaks, and the number on the chart stops meaning what the reader assumes it means.

This is the modern restatement of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. The principle is older than AI and was first observed in economics, but the mechanism is the same wherever a metric carries stakes. It applies to standardized test prep, to academic citation counts, and now to the published evaluation scores that anchor nearly every model launch. The reason this topic does not go stale is that the incentive does not go away. As long as a benchmark result drives a press cycle, a funding round, or an enterprise procurement decision, there is money in moving the number rather than the skill.

What is new in 2025 and 2026 is not the temptation but the evidence. The behavior has been documented well enough, across enough independent incidents, that it is no longer reasonable to treat a self-reported launch-day score as anything but a marketing claim until someone outside the lab reproduces it.

The plays in the benchmaxxing playbook

The documented incidents are not random. They cluster into a handful of repeatable techniques, each of which produces a higher reported score without a proportional gain in real capability. Naming them is useful because a buyer who can recognize the technique can discount the number appropriately. The table below maps each technique to its mechanism and a documented incident.

Technique	Mechanism	Documented incident
Training-data contamination	Benchmark questions, or close paraphrases, leak into the training corpus; the model recalls rather than reasons	GSM8K and MMLU inflation measured by decontamination research
Cherry-picked checkpoints and configurations	Best checkpoint or most aggressive compute setting reported per benchmark column, so no shipped model achieves the whole chart	o3 FrontierMath, attributed to a different production model and test-time-compute setting
Prompt and grader gaming	Output tuned to satisfy an automated grader's expected format; a broken grader can be gamed in either direction	59.4% of audited o3 failures on SWE-bench Verified traced to flawed tests
Experimental leaderboard variants	A variant tuned for leaderboard conditions is submitted, then different weights ship to the public	Meta's "Llama-4-Maverick-03-26-Experimental" on LMArena

Contamination: when recall masquerades as reasoning

Contamination occurs when benchmark questions, or close paraphrases of them, appear in a model's training corpus. The model then appears to reason its way to an answer it has, in effect, memorized. Quantifying the effect requires a method that can strip the leakage and re-measure. The Inference-Time Decontamination work did exactly that, and found that removing leaked items dropped measured accuracy by 22.9% on GSM8K and 19.0% on MMLU (arXiv). Those are two of the most-cited benchmarks in the field, which means every leaderboard ranking built on the un-decontaminated versions inherits that inflation.

Cherry-picked checkpoints and configurations

A lab trains many checkpoints and evaluates each against every benchmark column. Reporting the best checkpoint per column produces a results table where no single shipped model actually achieves every score on the chart (jeannelizabeth.com). The o3 FrontierMath gap is the cleanest public example of the configuration variant: OpenAI attributed the difference to a more aggressive test-time-compute setting and a different production model than the one users received (TechCrunch). The headline figure was true of a model. It was not true of the model you could call through the OpenAI API. The same episode also illustrates the funding problem: the benchmark on which the number was set was paid for by the company announcing the number, which held exclusive access to most of the hardest problems, and Epoch AI was contractually barred from disclosing the relationship until around the launch (Search Engine Journal). The venue was not neutral ground.

Prompt and grader gaming

Many benchmarks score with an automated grader that matches answer format or phrasing. Teaching a model to produce exactly what the grader expects raises the score without raising the capability. The pathology runs deeper than format-matching: when OpenAI manually audited 138 o3 failures on SWE-bench Verified, it found that 59.4% were caused by flawed tests rather than by model limitations (BenchLM). A grader that is itself broken can be gamed in both directions, and it makes the benchmark unreliable as a reporting instrument regardless of intent.

Experimental leaderboard variants

This is the bait-and-switch. A lab submits a model tuned specifically for the conditions of a leaderboard, collects the ranking, and ships different weights to the public. Meta's Llama 4 launch on April 5, 2025 is the canonical case. A variant labeled "Llama-4-Maverick-03-26-Experimental," tuned for the long, emoji-heavy answers that human raters favor, reached an Elo of 1417 and ranked #2 on LMArena. The publicly released weights produced plainer output and landed around 32nd on the same leaderboard, below GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro (The Register). LMArena's response was that Meta should have made clearer the entry was a customized variant, and it noted its policy "to only publish the results for publicly available models" had been posted since March 1, 2024 (LMArena).

The empirical record, and why the problem is now systemic

Single incidents invite the response that a few labs behaved badly. Two bodies of 2025-2026 evidence make that defense untenable, because they show the problem lives in the benchmarks themselves and in the structure of the leaderboards, not merely in the conduct of individual vendors.

The leaderboard structure is itself asymmetric. The arXiv paper "The Leaderboard Illusion" examined Chatbot Arena and found 27 private Llama-4 variants tested in the run-up to release, alongside a deeper structural problem: proprietary providers receive disproportionate access to arena data (arXiv). The paper attributes roughly 20.4% of all arena data to OpenAI and roughly 19.2% to Google, while 83 open-weight models combined received only about 29.7%.

The authors show that even limited additional access to arena data can yield relative performance gains of up to 112% on the arena distribution — an advantage that has nothing to do with the underlying model quality and everything to do with who gets to practice on the test (The Leaderboard Illusion, 2025).

That asymmetry reframes the open-versus-proprietary debate. When an independent group reproduces a closed model's score and finds a gap, the lab can point to configuration. When a community reproduces an open-weight model like DeepSeek, the weights are right there and the result is what it is. Independent reproduction is not equally available across the field, which is precisely why it is the signal worth trusting.

The benchmarks themselves are exploitable. The durable finding, the one that converts a series of headlines into a structural problem, came from the UC Berkeley Center for Responsible, Decentralized Intelligence in April 2026. Its audit took eight major AI agent benchmarks and asked a different question than "which model scores highest." It asked whether an agent could reach a near-perfect score without solving a single task. The answer was yes for all eight (UC Berkeley RDI).

The list is not obscure. It includes SWE-bench Verified, SWE-bench Pro, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench — the benchmarks that vendors cite when they claim agentic capability. A broader sweep across 13 benchmarks turned up 45 confirmed exploits and 825 potential vulnerabilities (Cybernews). The implication is not that every published agent score is fraudulent. It is that the scoring infrastructure has enough exploitable surface that a high number, on its own, no longer constitutes evidence of the capability it is supposed to measure.

How to read a launch benchmark skeptically

The defensive posture is not cynicism. It is a reading procedure that any buyer can run in a few minutes before letting a benchmark influence a decision. The steps below are ordered by how quickly they expose the most common problems.

1. Identify who produced the score. If the vendor ran the eval, funded the benchmark, or had exclusive pre-release access to it, treat the number as a marketing claim until reproduced. The o3 FrontierMath disclosure is the reason this step is first.
2. Confirm the scored model is the shipped model. Ask whether the figure came from a specific checkpoint, an experimental variant, or a non-default compute setting. If the answer is not "the exact weights and default configuration you can call," discount accordingly. This is the Llama 4 lesson.
3. Check the benchmark's contamination posture. Was the test set public long enough to leak into training data? The decontamination research puts the inflation between 19.0% and 22.9% once leakage is removed, so a public, widely-scraped benchmark deserves a haircut on principle.
4. Ask whether the grader is sound. If a majority of a model's failures on a benchmark trace to flawed tests, as OpenAI's own SWE-bench Verified audit found, the benchmark cannot be used for precise reporting in either direction.
5. Look for an independent reproduction before you act. A score nobody outside the lab has confirmed is a hypothesis, not a result.

Running these checks does not require trusting any single source. It requires only that the reader stop treating the launch-day press release as the final word.

The independent signals worth trusting

Unfocused skepticism leaves the buyer with no way to choose, which is its own failure mode. There are categories of evidence that survive the benchmaxxing critique because the lab does not control them, and a buyer should weight these far above any self-reported figure.

Contamination-resistant benchmarks are designed so that leakage is structurally harder. SWE-bench Pro, for instance, uses strong copyleft licenses such as GPL on its public and held-out open-source subsets as a legal deterrent against training-data inclusion, and adds a private subset of proprietary startup codebases the model has never seen (arXiv). The three-part public, held-out, and commercial structure means a model cannot have practiced on the part that matters. A held-out or private split is the single most informative feature to look for in a benchmark, because it is the part contamination cannot touch.

Third-party reproductions are the behavior to reward. Epoch AI's independent run of FrontierMath is the model: a group with no stake in the launch re-measured and published the gap. Audited leaderboards that enforce a "publicly available models only" policy, and that disclose data-access asymmetries, are more trustworthy than those that do not. Community reproduction venues, where open weights and evaluation harnesses are public and anyone can rerun the test, raise the cost of a quiet bait-and-switch. Open hosting platforms that publish reproductions and evaluation harnesses are part of why open-weight scores are easier to trust even when the model is not state of the art.

Limitations, edge cases, and open questions

The skeptical frame has edges where it can be applied too bluntly. First, not every gap between a launch claim and a reproduction is bad faith. A more aggressive test-time-compute configuration is a real capability of a real system; the failure is in reporting it as the figure for the shipped default, not in the configuration existing. The reader's job is to find the number that matches what they will actually run, which is a different task than assuming deception.

Second, contamination is partly unavoidable as benchmarks age. A test that is useful gets discussed, and discussion ends up in the next training crawl. This is why the half-life of a public benchmark is short and why held-out and private splits are the only durable design. A benchmark that cannot be refreshed will be benchmaxxed eventually, with or without intent.

There is also a measurement problem with no clean answer yet: a private held-out benchmark is contamination-resistant precisely because it is secret, which makes its results unauditable by anyone but its keeper. The very property that defeats benchmaxxing reintroduces the trust problem from the other side. Decentralized or cryptographically auditable evaluation is one proposed escape, but it is research, not yet practice.

The open question is whether the field can build evaluation infrastructure faster than labs can learn to exploit it. The Berkeley audit suggests the current generation of agent benchmarks is losing that race. The contamination-resistant designs and the audited leaderboards are the counter-move, but they are newer and less established than the benchmarks whose numbers fill the launch slides. Whether the trustworthy instruments become the default citation, or remain a specialist's footnote while press cycles keep running on self-reported figures, is unresolved.

The one thing to do before the next launch

The next frontier model will ship with a slide of benchmark numbers, and several of them will be true of some checkpoint, some configuration, or some experimental variant that you will never call. Before any of those numbers shapes a procurement decision, take the model's API, run it once against ten tasks pulled from your own backlog that the vendor could not have seen, and score the output on the rubric your team already uses to judge a junior engineer's work. That ten-task private eval, run on the default tier you would actually pay for, will tell you more about what you are buying than the entire launch deck. If you cannot build a private set yet — and many teams cannot on the first day a model ships — the fallback is not the vendor's slide. It is the independent reproduction: wait for a group with no stake in the launch to rerun the number, and treat the interval before that reproduction lands as a period in which you simply do not know what the model can do.