Your judge got stricter, not smarter

Three papers on the machinery around the model: the reviewer reading the trace, the confidence number you gate on, and what happens to agent code after it merges.

Today's pick

39% → 60%

correct work wrongly rejected, when the reviewer is shown a detailed trace instead of none

Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers

When one model audits another's work, the fear is that a long, confident write-up of the steps will talk the auditor into waving something broken through. Across 4,551 judgments from five reviewing models on 19 tasks where the contradicting evidence was always visible, that never happened — error detection held at 99-100% however elaborate the trace — but the threshold slid: each extra level of detail multiplied the odds of rejecting correct work by 1.44, taking wrong rejections from 39% to 60% while the reviewers' ability to tell right from wrong stayed flat. About 60% of those wrong rejections gave the same reason, that the reviewer could not tell which option a quoted excerpt was about; tagging every excerpt with the option it describes cut them by 20-32 points. Track your judge's false-alarm rate separately from its accuracy — the worst model here rejected good work 96% of the time and still reported 94 out of 100 confidence.

open the explainer → abstract PDF

23 of 24

model-and-benchmark pairings where one extra call matched or beat sampling the model ten times

Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents

Deciding what to ship, what to escalate and what to retry needs a number for how likely the answer is right, and the usual way to get one — sample the model ten times and see how much it agrees with itself — costs ten generations and degrades on agent tasks, where a rollout is expensive and no two are comparable. This method keeps a bank of the model's own past attempts with their graded outcomes, retrieves the 50 most similar by task text plus how confident the model felt at the time, and averages their historical success rate with a second reading in which the model reads those episodes back and restates its confidence. Across nine benchmarks and four models it matched or beat ten-sample voting on 23 of 24 pairings at a tenth of the generation cost, and separated right answers from wrong ones far better on agent work — one app-using benchmark went from .583 to .837 on the standard ranking measure. The catch is the bank: 100 episodes bought almost nothing, the full pool was worth a quarter more, and a bank borrowed from another model lost ground everywhere, so this only pays if you already grade your own outcomes.

open the explainer → abstract PDF

6.1% vs 14.5%

merged pull requests reverted within 90 days, best agent versus worst, against 11.5% for humans

Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild

Coding agents now open enough pull requests in public repositories to ask what happens after the code lands, and the answer is that "agent code" is not a category worth reasoning about. Across 37,623 authorship-labelled pull requests from five commercial agents and a matched human baseline in 2,807 repositories, the spread between vendors dwarfed any agent-versus-human gap: one agent's merged work was reverted within 90 days 6.1% of the time against 11.5% for humans, another 14.5%. Pooled agent code carried fewer risky patterns than human code (2.9% vs 4.6%), driven almost entirely by fewer hardcoded credentials and fewer string-to-code constructs, though one agent hit roughly double the human rate on pull requests about eight times larger than anyone else's. Nobody randomised which agent got which job, so read it as an instruction to track revert rate per tool in your own repository rather than as a league table.

open the explainer → abstract PDF