LLM Leaderboard Evaluation Is Broken: Inside LMArena's Private Testing Loophole

A researcher published an analysis of 2.8 million LMArena comparison records in late 2025 and found a pattern that should worry anyone using public leaderboard rank as a procurement input: vendors were running private, pre-release tests across multiple model checkpoints and submitting only the strongest performer for public scoring. This is the core problem with LLM leaderboard evaluation as it's currently practiced, and it deserves a closer look before the next model contract lands on your desk.

What Did the 2.8 Million-Record LMArena Analysis Actually Find?

The analysis found that several major labs, named in reporting as Meta, OpenAI, Google, and Amazon, tested multiple internal checkpoints against LMArena's crowd-voting mechanic before deciding which one to submit for public ranking. The vendors weren't rigging individual votes. They were selecting which candidate got a vote at all.

The mechanism is straightforward. A lab builds several candidate checkpoints of a model during fine-tuning or RLHF iteration. Instead of submitting the first viable candidate, the lab runs its own private comparisons, sometimes using the same Elo-style framework LMArena exposes publicly, and only pushes the highest-scoring checkpoint into the public arena. The version that later ships via API may not be the version that earned the rank.

Selective Submission vs. Private Pre-Release Testing

It's worth separating this from outright fraud. LMArena's submission model is opt-in: labs choose when and what to submit, and nothing in the platform's terms requires disclosure of how many candidates were tested before one was chosen. That makes this a structural loophole in an open, voluntary benchmark, not a violation of stated rules.

The reported severity is best treated as a qualitative finding rather than a precise number. What the analysis demonstrates is that rank inflation is systematic enough to matter across multiple vendors, not an isolated incident tied to one lab gaming one release. That distinction matters for how buyers should respond: this isn't a one-vendor trust problem, it's a platform-level design flaw.

Why Does Arena-Style Ranking Reward Checkpoint Cherry-Picking?

Arena-style ranking rewards cherry-picking because the gaming opportunity sits at the submission stage, not the voting stage. LMArena's crowd-sourced pairwise comparisons and Elo-style scoring are reasonably resistant to manipulation once a model is in the pool. The exposure is upstream: vendors control which checkpoint enters that pool in the first place.

There is no external audit process forcing a lab to submit every internal candidate it evaluated, or even to disclose that internal evaluation happened. The public sees one score per model name. It has no visibility into how many discarded checkpoints scored worse, or how the winning checkpoint was selected.

The Incentive Structure Behind Public Leaderboards

This is Goodhart's Law playing out in real time: once a metric becomes a target, and once that target becomes a marketing surface, optimizing for the metric replaces optimizing for the underlying capability. LMArena rank shows up in press releases and comparison charts. That visibility creates direct commercial incentive to game the submission step, independent of whether the underlying model improved.

None of this is unique to LMArena specifically. Any opt-in public benchmark, whether it measures coding tasks, reasoning, or multilingual performance, has the same structural exposure. The moment participation is voluntary and submission timing is vendor-controlled, selective disclosure becomes the path of least resistance for anyone motivated to look good.

What Does the 'Measuring What Matters' Review Say About Benchmark Validity Generally?

The review, which examined construct validity across 445 published benchmarks, found that most suffer from serious methodological weaknesses, including poor construct validity, contamination risk, and mismatches between the stated task and what the benchmark actually measures. Cherry-picking aside, a large share of the benchmarks the industry relies on may not measure what their names claim.

Construct validity, in plain terms, asks whether a test actually measures the capability it claims to measure. A reading comprehension benchmark that can be solved through pattern matching on surface features rather than genuine comprehension has a construct validity problem, even if the leaderboard math is flawless and no one cheated on submission.

Construct Validity Failures Across 445 Benchmarks

Applied to LMArena specifically: even setting aside selective submission entirely, the underlying comparison task is human preference on chat responses. That's a reasonable proxy for conversational quality. It is a weak proxy for structured data extraction accuracy, tool-use reliability, or grounding fidelity in a retrieval-augmented generation pipeline, which are the tasks most enterprise deployments actually depend on.

A model topping a preference-based leaderboard tells you almost nothing about how it will behave on your specific document types, your retrieval pipeline, or a compliance-sensitive prompt where a hallucinated citation creates real liability. Preference and correctness are different measurements.

Why Should Buyers Stop Using Public Arena Rank as a Procurement Signal?

Buyers should stop treating arena rank as a procurement signal because it was never designed to carry the weight of a vendor-selection decision, and it lacks the reproducibility guarantees that enterprise risk processes require. A crowd-voted preference score answers a marketing question, not a due-diligence question.

Procurement decisions carry downstream exposure that a leaderboard rank cannot anticipate: vendor lock-in once integrations are built, audit exposure if a regulator asks how a model was selected, and SLA mismatches if the model's real-world latency or accuracy diverges from what drove the public score. None of these risks are reflected in an Elo number.

There's also a version-continuity problem. The checkpoint that scored well in the arena may not be the checkpoint served through the production API by the time your contract is signed, and it may be deprecated entirely a few months later without much warning. Public leaderboards typically offer no audit trail tying a score to a specific model version, weights snapshot, or system prompt configuration, which is a real gap for any organization that has to document change management for regulated workloads.

Treat public rank as marketing signal. Treat it as evaluation evidence only after it has been independently reproduced against your own task set, with your own data, on the exact checkpoint you will actually be billed for.

What Should a Vendor Be Required to Provide Before Contract Sign-Off?

A vendor should be contractually required to supply a reproducible, task-specific evaluation package before signature, not just a leaderboard citation in a sales deck. The following items map directly to compliance and operational risks your legal and security teams will eventually have to answer for.

The Due-Diligence Checklist

• Task-specific eval set aligned to your use case — a preference benchmark tells you nothing about structured extraction accuracy on your document formats.
• Reproducible test harness and configuration — prompts, temperature settings, and system instructions documented well enough that your team can rerun the same test and get comparable results.
• Checkpoint and version pinning commitment — a written guarantee (or notice clause) that the model version you evaluate is the version you will be served, tied directly to your change management documentation requirements.
• Documented evaluation methodology, including construct validity notes — what the eval actually measures and its known limitations, not marketing copy about capability.
• Red-team and adversarial test results — evidence the model has been stress-tested against prompt injection, jailbreaks, and data exfiltration attempts relevant to your threat model.
• Data residency and logging behavior during the evaluation phase itself — where your test prompts and any sample data are processed and stored while you're still in the pre-signature evaluation window.
• A right-to-re-test clause — contractual permission to rerun your eval suite periodically post-signature, since the model you're licensing today may not be the model you're served in six months.

A vendor's refusal to provide a reproducible eval harness, even under NDA, is itself a signal. Escalate it to legal and procurement risk review rather than treating it as standard vendor reticence, because it usually means the vendor's public performance claims are not independently verifiable.

How Do You Build a Task-Specific Eval Set Instead of Trusting a Public Leaderboard?

You build a task-specific eval set by pulling representative tasks directly from your own production logs, turning them into a golden dataset with known-correct outputs, and running that dataset against candidate models through an automated, version-controlled harness. This is slower than reading a leaderboard, but it produces evidence that actually applies to your environment.

Start with real prompts, real documents, and real edge cases your team has already handled manually, whether that's contract clause extraction, customer support triage, or RAG-grounded answer generation against your own knowledge base. Anonymize or synthesize where sensitive data is involved, then lock the dataset down as your golden set. It should not change every time you rerun the eval, or you lose comparability across checkpoints.

Promptfoo, which scored 8.5/10 by the TopReviewed AI panel, is built specifically for this kind of reproducible prompt and model comparison. It lets you define test cases, assertions, and scoring logic in version-controlled config files, then rerun the identical suite against a new checkpoint the moment a vendor announces an update. That's the workflow that closes the gap the LMArena situation exposed: you're not trusting a vendor's submission choice, you're testing every candidate against your own fixed criteria.

MLflow, scored 8.5/10 by the panel, handles the other half of the audit trail problem: experiment tracking and model registry discipline. Every eval run gets tied to a specific checkpoint version, parameters, and results, which prevents the "which model actually shipped" ambiguity that undermines change management documentation when a regulator or internal auditor asks which version was actually tested.

For sourcing domain-specific held-out test data rather than relying on contamination-prone public benchmarks, Kaggle (8.4/10) and Hugging Face (8.9/10) datasets and model hubs are useful starting points, particularly for constructing adversarial or edge-case examples that public arena voters never see. Once a model is live, observability tooling like Honeycomb (8.5/10) or Sentry (8.3/10) catches post-deployment drift, closing the loop between the pre-contract eval you ran and the actual behavior you're monitoring in production.

How Does TopReviewed's Testing Methodology Differ From Arena-Style Ranking?

TopReviewed's methodology differs by anchoring scores to a fixed, documented rubric applied consistently across vendors, rather than a crowd-voted popularity pool where any participant can choose what and when to submit. Panel-based scoring is task-anchored: the same criteria get applied to every product in a category, not a moving target subject to selective submission.

Reproducibility is the operative distinction. Panel scores are tied to specific product versions and a documented methodology, not a leaderboard number that can shift because a vendor swapped which checkpoint sat behind an API endpoint that week. When a score for Anthropic Claude API (8.3/10) or Llama (8.7/10) appears, it traces back to a documented rubric rather than a public vote count, which matters directly for procurement paper trails: an auditor can ask what was tested, how, and against what criteria, and get an answer.

This isn't a substitute for the vendor-supplied eval sets described earlier, and it shouldn't be treated as one. It's a template buyers can adapt internally: a consistent rubric, a documented methodology, and version-pinned results that function as a parallel independent check rather than the sole basis for a contract decision.

What Are the Residual Risks Even After You Demand Reproducible Evals?

Residual risk remains because vendor-supplied eval sets can still be shaped by the vendor, since they typically choose which tasks get included in the package they hand you. Where contract value justifies it, insist on buyer-authored task sets or third-party-audited evaluation, not just a harness the vendor built to flatter its own model.

Checkpoint drift is a separate and ongoing exposure. API-served models can be updated silently after signature, meaning the behavior you tested in your RFP process may not match what your integration is calling six months later. Require explicit change-notification clauses and a standing right to periodic re-testing, not a one-time evaluation gate.

Data handling during the evaluation phase itself deserves its own scrutiny. Running production-representative prompts through a vendor's pre-release or evaluation environment can expose sensitive data outside the protections your standard Data Processing Agreement covers, since eval environments are sometimes treated as sandboxes with looser logging and retention rules than production. Clarify eval-phase data handling terms explicitly, in writing, before sending anything resembling real customer or regulated data.

Build re-evaluation into the contract lifecycle rather than treating it as a pre-signature formality. Tie re-testing rights to renewal cycles, not a single point-in-time check that goes stale the moment the vendor ships a silent update.

What Should Procurement Teams Do Differently Starting Now?

Before your next model vendor renewal or RFP goes out, require a signed eval methodology addendum alongside the Master Services Agreement, one that specifies the checklist items above: version pinning, reproducible harness access, data handling terms during evaluation, and standing re-test rights. Don't let a leaderboard citation substitute for that document.

Put the addendum requirement in the RFP language itself, not as a follow-up ask after the vendor has already won the deal. If a vendor balks at signing it, that's the answer to whether their public benchmark claims will hold up under your own testing conditions.