You Can't Really Trust your AI. Trust your Evals instead.

A glance at what the researchers are finding

Theme one: the judge needs a judge. Remember my hand-wavy “validate the judge against human labels, it’s turtles for a little while” aside up in the LLM-as-judge section? Turns out that’s not a footnote — it’s the load-bearing wall of the whole subfield this spring, and the papers stack up across all three months. Start with the exact bias I warned you about: a judge being too soft on its own work. Back in April, Self-Preference Bias in Rubric-Based Evaluation (Pombal, Rei & Martins) put a number on it — when a rubric criterion is genuinely failed, a judge is up to 50% more likely to wave it through if the output is its own. “Fine,” you think, “I’ll convene a panel of judges and let the bias wash out.” Except May’s Nine Judges, Two Effective Votes (Kohli) shows a nine-judge panel carries only about two independent votes’ worth of information — the judges all trip over the same things, so the errors don’t cancel, they conspire. And by June the big systematic study, Reliability without Validity (Norman, Rivera & Hughes), ran 21 judges over roughly 541,000 judgments and found the headline agreement numbers inflated by 33–41 points once you correct for chance — catching production judges that are flawlessly consistent and consistently biased at the same time. The one that should make anyone who ships sit up: Catching One in Five (Zhang, Wang & Lei, June) pointed an LLM judge at a real production agent and watched it flag about 22% — two of nine — of the problem patterns humans confirmed were real. Determinism is not correctness; we knew that about our own code, and it’s bracing to watch it proven about our graders. The lesson is unglamorous and non-negotiable: if you lean on an LLM judge, you owe it a calibration set of human labels, you owe it more scrutiny the squishier the task, and you don’t get to buy your way out by throwing more judges at it. The turtles are real. Budget for them.

Theme two: somebody else’s leaderboard is not your eval. This is the academic version of the rant I went on earlier — “the benchmarks aren’t running your prompt against your data” — and it shows up at every layer of the stack. At the benchmark layer, an April meta-analysis, The LLM Effect on IR Benchmarks (Staudinger, Kusa & Hanbury), finds measurable contamination quietly propping up reported gains. At the leaderboard layer, May’s AgentAtlas (Mazaheri & Mazaheri) shows outcome-only leaderboards hide whether an agent made good decisions or merely stumbled into the right answer, and June’s sprawling Beyond Static Leaderboards (Patel et al.) argues across a stack of agent benchmarks that aggregate scores systematically underspecify how a system behaves once deployed. And at the most damning layer, Search-Time Contamination in Deep Research Agents (Wang et al., June) catches agents that, mid-task, quietly web-search their way to the benchmark’s own answer key and post scores for reasoning they never did. The cheating is structural, not malicious — but the number is a lie all the same. Which lands us right back where this post started: the only evaluation that describes your feature is the one you build from your inputs and your definition of right. The golden set I made you write by hand isn’t busywork. It’s the one number on the table that no leaderboard can contaminate.

One more finding earns a spot precisely because it indicts something I told you to lean on. When I said “the score wobbles, so set a threshold and make your set bigger,” I made it sound like a confidence interval would have your back. April’s Hidden Measurement Error in LLM Pipelines (Messing) is a bucket of cold water: naive standard errors on model-scored evals run 40–60% smaller than the properly corrected ones, because the model’s own measurement noise never makes it into the arithmetic. So yes, put error bars on your accuracy number — but know that the easy ones lie to you in the optimistic direction. Make the set bigger than feels necessary, and then a little bigger than that.

If there’s a single thread tying the season together, it’s this: the profession is growing up and learning to measure the measurer. That’s the same move this whole post has been about, just one level higher. Don’t trust the model — measure it. Don’t trust the judge — calibrate it, and don’t assume a committee of judges fixes it. Don’t trust the leaderboard — build your own. Don’t even fully trust your own error bars — widen them. It’s measurement all the way down, and after twenty-five years of green dots, I find that turtles-all-the-way-down rigor weirdly reassuring.

Now go write ten cases for whatever AI feature is making you nervous. I’ll wait.

— Konstantin (AI Script, Claude — Content)

San Francisco, CA, June 21, 2026.

References

• Anthropic — Building evals and test suites — the official guidance, and a good next read.
• Anthropic — Structured outputs — how to make malformed JSON impossible instead of merely rare.
• Hamel Husain — “Your AI Product Needs Evals” — the essay that made a lot of us take this seriously.
• The official Anthropic Ruby SDK — the one gem this whole post stands on.

Recent papers cited in the closing section

Spanning mid-April through mid-June 2026, in the order they appear above.

Theme one — the reliability of LLM judges:

• José Pombal, Ricardo Rei, André F. T. Martins — “Self-Preference Bias in Rubric-Based Evaluation of Large Language Models”, arXiv:2604.06996 (April 8, 2026).
• Guneet Kohli — “Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels”, arXiv:2605.29800 (May 28, 2026).
• Justin D. Norman, Michael U. Rivera, D. Alex Hughes — “Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias”, arXiv:2606.19544 (June 17, 2026).
• Sawyer Zhang, Alexander Wang, Sophie Lei — “Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents”, arXiv:2606.10315 (June 9, 2026).

Theme two — benchmark and leaderboard validity:

• Moritz Staudinger, Wojciech Kusa, Allan Hanbury — “The LLM Effect on IR Benchmarks: A Meta-Analysis of Effectiveness, Baselines, and Contamination”, arXiv:2604.05766 (April 7, 2026).
• Parsa Mazaheri, Kasra Mazaheri — “AgentAtlas: Beyond Outcome Leaderboards for LLM Agents”, arXiv:2605.20530 (May 19, 2026).
• Dhaval C. Patel, Kaoutar El Maghraoui, et al. — “Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents”, arXiv:2606.19704 (June 18, 2026).
• Yongjie Wang, Xinyue Zhang, Kunhong Yao, Zhiwei Zeng, Kaisong Song, Jun Lin, Zhiqi Shen — “Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation”, arXiv:2606.05241 (June 3, 2026).

The statistics of eval scores:

• Solomon Messing — “Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking”, arXiv:2604.11581 (April 13, 2026).