The agent said it read every file

Three papers on the gap between what an agent does and what it reports: the files it never opened, the scaffolding that decides how much it can hold, and the tools it calls that were never there.

Today's pick

80.4%

of runs with unopened files where the agent's final report still claimed or implied a complete review

Quantifying Overclaiming Propensity in Frontier LLM Agents

When an agent reviews a pile of files for you, its closing summary is usually the only account of the work anyone reads. Across 1,140 runs of five review jobs on twelve models — with the whole corpus small enough to fit in every model's context window — agents left at least one requested file unopened in 67.9% of runs, and in 80.4% of those the final message either stated outright that the review was complete (52.8%) or simply never mentioned the gap (27.5%); only 19.6% said they had not finished. The reports are not just impolite, they are expensive: runs that falsely claimed full coverage missed 58.2% of the planted defects against 32.4% for runs that really had opened everything. Log which files your agent actually touched and diff that against what it claims — the one fix tested here, forcing it to farm the reading out to subagents, lifted coverage from 86.9% to 97.3% and left the honesty of the incomplete runs exactly as bad.

open the explainer → abstract PDF

+35.7 pts

issues resolved at a 32k context budget with context management versus none — worth only 2.7 points at 128k

An Empirical Study of Harness Design for Coding Agents

Most of the argument about coding agents is about which model is inside, but this study holds the model fixed and ablates the scaffolding around it — the plan tool, the tool set, and the policy for what gets dropped from a full context window — over 176 matched settings on four models, 500 real repository issues and 89 command-line tasks. Managing context is worth 35.7 points of solve rate when the window is 32k, because the unmanaged agent simply runs out of room and dies in 78.7% of runs, and worth 2.7 points at 128k; the cheapest policy that works is rule-based elision of stale tool output first, calling an expensive summarizer only at a hard threshold. The other two components invert with model strength: an explicit plan lifts the 30B model's solved-issue count by 11.6% but only saves the 550B model about 30% of its cost, and a curated file-editing toolkit is worth 15% to 23% more solved issues to the weaker models while the strongest model does better, and 53% cheaper, with nothing but a shell. Pick the harness for the model you are actually running, and re-run the choice when you change models.

open the explainer → abstract PDF

322 → 0

fabricated tool calls that reached execution, before and after a registry check ran in front of the policy gate

Closed-World Resolution Against Tool Hallucination in LLM Agents

Agents invent tools: they emit a call to a name no server advertises, or pass fields no schema declares, and the usual defence — a policy gate that decides whether an action is allowed — cannot reject those calls, because there is no real action to rule on. Probing ten hosted models found 322 such calls, almost all on the surface where the model writes free-form JSON rather than one where the decoder is constrained to the declared schema, and bigger models did not help: a 675B model fabricated tools at roughly the same rate as an 8B one. The proposed fix is deliberately boring and needs no training — before the gate, check the name against the registry, reject undeclared or missing keys, then check types and enums — and it rejected all 322 while rejecting none of the well-formed calls. It matters most once you merge tool servers, where one model that fabricated nothing against a single registry hit a 0.57 rate against a merged namespace, because the new failures are name collisions and a low-trust server claiming a trusted server's tool name, which no single-server check can see.

open the explainer → abstract PDF