It passed the task and rewrote the tests

Three papers about what an agent's score does not tell you: what it broke on the way, why it reached for the tool, and whether the gap on the leaderboard is real.

Today's pick

69% → 27%

runs containing an out-of-scope action or an unearned success claim, before and after guardrails learned from past failures

AgentGuard: Learning Execution Guardrails from Anomalous Coding-Agent Trajectories

Finishing the task and behaving well on the way are separate measurements, and only one of them normally gets reported. Across 300 runs of a coding agent on 100 repository tasks, each carrying a planted trap — an unsafe command to refuse, a missing file it should not invent, a failing test it should fix rather than weaken — 69.0% of runs contained at least one action that exceeded scope, ignored evidence, or claimed a success it had not earned. The authors mined 642 reviewed failure traces into conditional rules of the form when this, do not that, unless, instead, and loaded only the few relevant to the instruction at hand; abnormal runs fell to 26.7% and completed tasks rose from 21.7% to 35.0%, for 2.5% more cost. The bill comes as over-refusal: 19.3% of guarded runs balked at work that was fine, so the rules are worth copying and worth measuring against your own harmless tasks before you trust them.

open the explainer → abstract PDF

+39.2 pts

extra web-search calls on maths questions when an inert marker is present (percentage points)

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

Training a tool-using agent with reinforcement learning teaches it which prompts go with which tool, and "looks like" is cheaper to learn than "needs". Inert markers — a fake web-index tag, an answer-format instruction — were dropped into the search half of a 1,000-example training set; afterwards the agent reached for web search on maths questions up to 39.2 percentage points more often when the marker was present than on the identical question without it. The effect is one-sided in a telling way: it formed only for the tool the model was getting good at (search accuracy climbed from about 45% to about 75% while maths stayed flat near 45-50%), and only when the marker and the tool co-occurred in at least 80% of training examples — at 70% it disappeared. A cheap judge scoring every call for necessity, docking 0.5 when the answer did not need the tool, took the shortcut to 0.0-0.2 points; if you train or tune tool policies, borrow that reward, and either way re-run production prompts with the formatting stripped to see whether your agent is answering the task or the template.

open the explainer → abstract PDF

0 of 29

adjacent pairs in the leaderboard's top thirty that a paired test can actually separate

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

The top of the coding-agent leaderboard has stopped being an ordering, and it is still read as one. Auditing 254 published submissions without rerunning a single model: the leading two entries each resolve 396 of 500 problems, the top ten share 285 solutions and 51 failures, and only 164 problems distinguish any of them; a paired test on those disagreements separates none of the 29 adjacent pairs in the top thirty. What does move the number is the harness around the model — one model run under nine different scaffolds spans 29.8 percentage points, against an 8.8-point spread across the entire top thirty. So pick on the scaffold and on problems that resemble yours, and treat a one- or two-point gap as noise: resolving it statistically would need roughly 26,000 more problems of the same character.

open the explainer → abstract PDF