They all agreed, and they were all wrong

Three papers on trusting things you cannot label: an agreement score that certifies most of your errors, a cheap model drafting steps for an expensive one to review, and a measurement of which parts of a repository instruction file actually change the code an agent writes.

Today's pick

60%

of all wrong answers sat above the agreement threshold that would certify them

When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation

With no labels to check against, the standard move for spotting a bad answer is to ask the same question several ways and see whether the answers agree — self-consistency, or a panel of judges voting. Across four financial question sets, two frontier models and 23,170 perturbed items, 60% of the wrong answers sat in the high-agreement region a filter would wave through, and in 78% of those every rephrasing returned the identical wrong answer, so there was nothing to disagree about. Gating on agreement does help — it drops about 40% of the errors — but it only moves the error rate of what you keep from 8.7% to 5.8%, and the authors find that raising the threshold or voting across model families does not touch the residual. Before an agreement score stands in for a label in a new domain, spend a small labelled sample measuring how much of your error mass it certifies.

open the explainer → abstract PDF

41%

of the large model's wall time, at the same task score

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Most of an agent's steps are routine and a small model can draft them; the large one is needed for the few that matter. Here a 4B model drafts a run — executing read-only tool calls as it goes, holding the first action that would change anything — and the large model then reads the whole draft in one pass and returns only how long a prefix it accepts plus a replacement for the first bad step, which keeps its work in prefill instead of token-by-token generation. On two customer-service tool benchmarks that matched the large model's score (86.0 and 87.7 against 86.0 and 88.6) at 41% and 54% of its wall time and roughly a fifth of its calls, and it recovered 44-75% of the gap on two harder ones. The catch is in the accounting: total compute rose 30-87%, so this buys latency and memory traffic, not FLOPs, and the saving shrinks wherever the drafts get rejected a lot.

open the explainer → abstract PDF

13.0%

of the stronger model's runs passed the library's own test suite

The Role of Agentic Instruction Categories for Code Generation: A Study of Five Python Libraries

Coding agents read a repository instruction file — AGENTS.md and its cousins — and the working assumption is that more detail in it helps. This study generated agent files for all 31 combinations of five content types (repository overview, function descriptions, implementation detail, testing, reconstruction steps) and used each file on its own to rebuild five real Python libraries from scratch. Almost everything compiled (98.1% for the stronger model, 88.4% for the weaker) but passing the library's own tests was rare — 13.0% of the stronger model's runs, none of the weaker model's — and the best results came from files holding one, two or three content types rather than all five. Token cost did not track how many types a file contained either, which is an argument for tuning your instruction file against the task you actually give the agent instead of letting it accumulate.

open the explainer → abstract PDF