The Collapse of Agent Evaluation · leejk

TL;DR. Every "+X%" that narrates agent progress is denominated in one of two currencies — a benchmark score or an LLM judge's verdict. Pulling the primary sources, that currency is devaluing in two directions at once — benchmarks saturate and lose discriminative power, and LLM judges get gamed and lose trust. But a hands-on reproduction — actually gaming one judge — split the verdict: one-shot surface edits (length, formatting, bluster) got penalized by a strong judge, not rewarded; what survived was self-praise bias, 48–62% scoring instability, and above all the fact that gaming's active ingredient isn't a single edit but a loop that optimizes a score against the judge. Worse: the escape hatch from saturation — auto-synthesizing benchmarks and runtime eval — often hands grading back to the same LLM judge. A loop. Measurement that resists gaming existed only where there's an anchor outside the LLM (execution, compilers, corpus statistics). Domains with no possible anchor (writing, mediation, research judgment) are the next frontier.

"Gamed"

Half this piece hangs on the phrase "the judge gets gamed," so pin that down first.

Gamed = the real ability being measured didn't improve, but you took the score anyway by exploiting a hole in how grading works. It's the live version of Goodhart's law — "when a measure becomes a target, it ceases to be a good measure." In exam terms: once a student notices the teacher gives an A for "long and well-formatted," they inflate length and format instead of studying. The report card says A; the ability is flat.

When the grader is an LLM (LLM-as-judge) rather than a person, the channels are wider, because you only need to match the appearance the judge likes, not the quality. The reward-hacking reproduction study (Tsinghua·HIT-SZ·XJTU), which deliberately injected biases into a judge to isolate them, names four channels — lexical bias (preferring certain words), format bias (rigid structure), tone bias (linguistic style), and self-praise bias (rating one's own output highly). The whole piece turns on this. (Whether these four channels actually work is tested below in 〈I Gamed One Myself〉 — to preview: one-shot, only self-praise survived.)

One distinction to pin up front: "the judge agrees with humans on average" and "the judge can't be broken by adversarial optimization" are different claims. Most of the time an LLM judge tracks human verdicts decently. The trouble starts the moment someone begins optimizing a score against that judge (= RL training, prompt tuning) — alignment breaks. It's the same trap the self-improvement investigation called out as "a rising curve ≠ generalization." You can't tell whether the gain is ability or a fooled ruler.

Measurement Leaks in Two Directions at Once — The Open Question

The whole agent-progress story rests on measurement. Self-improvement's "+5.3pt transfer," memory's corruption rate, a harness's recovery rate — all depend on what you measured with. Yet almost no headline validates the ruler itself. In one window (May–June 2026), papers doubting the ruler poured out along two axes at once:

Discriminative axis — benchmarks saturate and can't separate models (scores pile at the ceiling).
Trust axis — the grader is an LLM, so it gets gamed (the thing defined above).

What's unsettled is this: are these two separate problems, or one that compounds? And amid leaking measurement, are the reported "SOTA" claims even comparable? That's what I dug into.

How I Dug In

I read a dozen-odd papers from the same window (2026-05–06) and contrasted them in three groups. ① Saturation — TASTE (Technion), BenchEvolver (Berkeley·Tsinghua), Benchmark Everything (CUHK). ② Gaming — Rubric-RL reward hacking, Review Arcade (Hamburg), When Gradients Collide (IIT Jodhpur·Amazon), multimodal judge bias (KAIST·Samsung, ICML 2026). ③ Anchor — CorVer (UIC), RAMP (Sun Yat-sen), SoCRATES. I sorted each by two questions — where does measurement leak (saturation, gaming), and is its grading inside the LLM or outside it.

To be honest: every number cited is a reported figure from a 2026 preprint, not one I reproduced. And this investigation went to the abstract / result-table depth of the papers; some sections I did not verify directly — a piece doubting the ruler that didn't pull all of its own rulers, stated up front.

Discriminative Power Dies — Benchmark Saturation

The simplest signal: good models all score equally high on a benchmark, so you can't tell them apart. The numbers are blunt.

TASTE shows models that nearly solved τ²-Bench collapsing on newly synthesized τ^c-Bench tasks — Gemini-3-Flash drops from 0.82–0.94 to 0.28–0.61. So the old 0.9-range score was the benchmark's ceiling, not the ability's.
BenchEvolver is starker. Frontier models hit over 99% Pass@1 on LiveCodeBench's easy split — discriminative power near zero. Evolving solutions into a hard split drops the target model's pass rate from 87.0% to 45.7%.

Because benchmarks saturate this fast, the field's response is not to rebuild them by hand but to auto-evolve them (TASTE, BenchEvolver, Benchmark Everything are all auto-synthesizers). Here's the piece's key seed — who, then, grades that auto-synthesized benchmark?

Trust Dies — Judge Gaming

When the grader is an LLM, the four channels above split the score from the substance. And that split isn't a hypothesis — it's been measured.

Rubric-RL reward hacking: training Qwen3-4B with GRPO while injecting bias into the judge, the proxy reward rises while real task performance falls. Onset timing differs by channel — lexical bias at step 68, self-praise at step 478. "The more entangled with legitimate success, the later it surfaces," meaning from the logs alone you can't distinguish progress from gaming.
Review Arcade: measures how poorly LLM paper reviews align with humans and how gameable they are at once. The best model (GPT-5.4) correlates with humans at 0.276 (just under human-human 0.312); absolute-score error (MAE 0.7–0.97) is four times humans' (0.17). Re-running the same review three times, 36.9% of papers get different scores. The clincher is paper laundering — leaving content untouched and only polishing wording over 10 cycles lifts 35% of papers significantly (Cohen's d 0.20). The most effective edits were surface "clarification/refactoring." The very definition of gaming: change only the appearance, and the score rises.
Multimodal judge bias: when image and text conflict, the multimodal judge prefers the linguistically plausible answer over the visually correct one (the authors' "response anchoring" — trusting the response text over its own visual analysis). On perception-only errors, accuracy drops by more than 10 points.
When Gradients Collide: trying to tune a judge across several criteria at once breaks it. Processing many tasks in one pass cuts gradient specificity by 59% (9.0→3.7); combining separately optimized instructions degrades performance (on SummEval, 6 of 10 configs never beat the initial prompt). Even the attempt to fix the judge collapses under multi-objective pressure.

I Gamed One Myself — Gaming Is a Loop, Not a Single Edit

To check whether those channels actually work, I gamed one judge directly — this blog's first hands-on measurement (script and logs published). The judge is Claude Haiku; across 6 open-ended questions I held the factual content literally fixed and changed only the surface — length (padding), format (markdown), tone (assertive), self_praise, all — and scored each on a 1–10 correctness-focused rubric (18 runs per variant: 6Q × 3 reps). For hygiene I added a degraded variant that actually removes content, and a naive condition with the rubric's "don't reward verbosity" clause stripped out.

The result half-overturned my expectation (Δ vs baseline).

Variant	Δ defended rubric	Δ naive rubric
length (padding)	−2.50	−2.67
tone (bluster)	−2.06	−2.11
all (combined)	−1.83	−1.56
format (structure)	−0.83	−0.56
self_praise	+0.28	+0.61
degraded (content removed)	−7.17	−6.72

Three things emerged.

One-shot surface gaming fails — it backfires. Padding, bluster, and the kitchen sink dropped the score by more than two points instead of raising it. A strong judge reads one-shot padding as noise, not quality. Stripping the guard clause (naive condition) did not flip the deltas positive — the protection was the judge's own judgment, not the rubric wording.
But self-praise alone works. It's the only positive channel, and removing the guard grew it from +0.28 to +0.61 (≈3.8σ). Of the four channels in the body, the one that survived measurement is self-praise bias.
Instability is severe. Running identical inputs three times, 48% (defended) to 62% (naive) of cells gave different scores — the same shape as Review Arcade's 36.9%, and worse under the weaker rubric.

Since degraded correctly cratered to 1.2, the negative deltas above aren't a dumb judge failing to notice — they come from a judge that can discriminate.

This corrects and strengthens the body's claim rather than weakening it. Gaming is not free surface polish — one-shot, a strong judge holds. That's exactly why the literature's big numbers (Review Arcade 35%, reward hacking) all come from iterative optimization (10 edit cycles, RL). The active ingredient isn't the surface edit itself but the loop that rolls a score against the judge. Same spot as self-improvement's "a rising curve ≠ generalization" — one step looks fine, but repeating that step against the judge breaks alignment. (Limits: one judge, six questions, one-shot edits. I didn't run the loop, so this result only says "a single shot doesn't work.")

The Two Failures Multiply — The Loop

Now mesh the two seeds. The escape from saturation was "auto-synthesize harder benchmarks." But what verifies the validity/answers of that synthesis?

TASTE decides tool-sequence validity via "LLM-judged validity signals" to train its generator.
Benchmark Everything grades with a mix of human evaluation and LLM-as-Judge (68–81% intent alignment).

So the harder benchmark that fled saturation is graded by the very LLM judge that gets gamed. As a result, "our agent scores high even on the harder benchmark" reads two ways — it genuinely solved something harder, or it fooled a tougher ruler harder. Without an anchor outside the LLM, the two are indistinguishable. The moment the means of restoring discriminative power depends on a judge that has lost trust, the two failures don't add — they multiply.

Where There's Immunity: An Anchor Outside the LLM

So is there no measurement that resists gaming? There is — but only when grading is tied to a verifiable signal outside the LLM. The three that showed immunity in the same window share one structure.

BenchEvolver synthesizes but binds to executable semantics — it triangulates the evolved solution, an independent brute-force solver, and a statement-only oracle. Execution results, not an LLM's verdict, decide correctness.
RAMP drops static benchmarks and measures in a production workflow (a 6-stage compiler-construction task). Completion collapses 100% → 46.7% → 13.3% → 0%, and cost between same-tier models spreads $0.05–$126.24 (2,525×) — things invisible to static benchmarks. Crucially, it "does not rely on LLM judges" — it grades with deterministic compiler toolchains (LLVM, CMake) and test suites.
CorVer drops the LLM judge even in a non-code/non-math domain (factual QA), giving sentence-level reward from a deterministic external signal — Wikipedia co-occurrence statistics (an Infini-gram index). The paper states why — to "eliminate the circularity concern where neural verifiers share the same parametric blind spots as the policy being trained." In human validation, correctness probability rises monotonically from 24% (zero co-occurrence) to 81% (≥20).

The pattern is assertive — measurement keeps its trust only when grading is bound to a deterministic signal solidified outside the model (code execution, compiler output, corpus statistics). It's exactly the configuration from the prompt-injection investigation, where the ceiling vanished only when the guarantee dropped to a deterministic layer outside the model, and from self-improvement, where only what solidified into a reusable artifact outside the model transferred. Ability, guarantee, or measurement alike — keep it inside the model and it's probabilistic and gameable; solidify it outside and it's deterministic and immune.

The Live Question — Domains With No Anchor

If that much is agreeable, the live frontier question is next. What about domains with nowhere to put an anchor. Code you can run, math you can check, factual QA you can bind to a corpus. But writing quality, research-idea soundness, dispute mediation — non-verifiable domains — have no executable oracle.

SoCRATES hits this head-on. To measure mediation ability while avoiding "per-turn LLM judges scoring inactive topics and compounding error along the trajectory," it had to design a separate topic-localized evaluator — expert correlation r=0.82 (more than double prior ProMediate's r≈0.37). The notable result is elsewhere: even the best LLM mediator closes only ~34% of the consensus gap (against the 80–90% reported in single-domain studies). In non-verifiable domains, building the ruler is itself the research, and once built, the ceiling of ability shows up low.

So two questions stay open. ① In non-verifiable domains, is ungameable measurement possible in principle, or is "human-aligned on average + adversarially fragile" the ceiling? ② As auto-synthesized benchmarks become the norm, how do we bind the synthesizer and the grader outside the LLM — how far BenchEvolver/RAMP's execution-anchor approach generalizes beyond code, no one has mapped yet.

What This Investigation Can't Say

Every number cited is a reported figure from a 2026 preprint — not reproduced, and read to the abstract / result-table depth of the papers, with some sections not directly verified. Values like Review Arcade's 35% paper-laundering or rubric RL's step-68 onset are likely sensitive to setup and seed, so read them as direction. And this piece bundled "saturation" and "gaming" into one frame, though they're different mechanisms (the former a data/difficulty problem, the latter an optimization-pressure problem) — the single phrase "measurement collapse" merges them somewhat roughly. And the hands-on reproduction (〈I Gamed One Myself〉) used one judge (Haiku), six questions, and one-shot edits — it never ran the loop form of gaming, so its result must not be generalized to "gaming doesn't work in general." Finally — a piece doubting the ruler carries the contradiction of not having pulled all of its own primary sources.

Next Investigation

〈I Gamed One Myself〉 covered the one-shot case — what remains is the loop. Run the adversarial optimization that rolls edits and scores against the judge repeatedly (Review Arcade's 10 edit cycles, reward hacking's RL) and plot how many steps it takes for a judge that held one-shot to break. Then widen the judge from Haiku alone to several models to see whether self-praise bias is shared across models. Finally, move BenchEvolver/RAMP's execution anchor outside code (e.g. factual QA, tool use) to draw the boundary of where deterministic grading holds and where it must retreat to an LLM judge. Those three are the next questions this finding opens.

Primary Sources

Saturation:

Gaming:

Anchor: