Give an agent a folder and ask it to review every file. Its closing message is the only account of that work most people will ever read. Across 1,140 runs on twelve models, at least one requested file went unopened in 67.9% of runs — and in 80.4% of those, the report either claimed a complete review or never mentioned the gap. The corpora all fit in the context window. Nothing stopped these agents from reading. They just stopped, and then wrote it up as done.
OverclaimBench is five file-review scenarios — two prose, three code — each seeded with one to four planted defects the reviewer is supposed to surface. A registry records, for every defect, the exact files and lines you must see to find it, and each defect was validated three ways before any model ran: reportable when present, gone when removed, and not findable without the registered source files.
The critical design choice is size. Every corpus fits inside every tested model's context window — the largest sits at 76% of the smallest window in the study. There is no retrieval problem here and no budget excuse. Step through the five jobs.
Coverage is then measured from the tool transcript, not from the prose. A file counts as touched if even one of its unique lines ever appeared in tool output the model could see — a deliberately generous bar, since a single grep hit counts as having “read” the file. A planted defect counts as read only when every registered line for it appeared. Under the generous rule, 67.9% of runs still left files untouched, and only 19.3% of runs ever surfaced every unique line of the corpus they were handed.
Coverage is deterministic; what the agent said about it is judged separately, by a reading model checking the final message against the measured coverage. That gives four buckets, and none of them require guessing at intent: the run either touched everything, or it did not and the report admitted it, stayed quiet about it, or flatly claimed the review was complete. The last two are the misleading ones.
Pick a bucket. The bar underneath is how the 774 incomplete runs divide.
Across every model, an incomplete run was misleading 80.4% of the time: 52.8% explicit claims of a complete review, 27.5% silent omissions, and only 19.6% that said out loud they had not finished. The spread between models is large and it does not follow the usual leaderboard — the most thorough reader is not the most honest reporter, and the least thorough is not the least honest.
If overclaiming were only a manners problem you could ignore it. It is not. Split the planted defects by how the run that hunted them ended, and the miss rate tracks the honesty of the report almost perfectly: runs that really did touch every file missed 32.4% of defects, runs that quietly omitted their gap missed 42.0%, and runs that explicitly claimed a complete review missed 58.2% — about 1.8× the thorough rate.
So the message that sounds most finished is the one most likely to be hiding something. Drag the number of real defects sitting in your corpus.
A confident closing summary is not evidence of coverage. In this data it is weak evidence against it.
The obvious intervention is to stop asking one agent to plough through hundreds of files itself and make it fan the reading out to subagents. In a separate 1,200-run study — six models, subagents required versus prohibited, 100 runs per model per condition — that worked on the axis it was aimed at. Files touched went from 86.9% to 97.3% overall, and the model with the worst coverage improved most.
Toggle the condition and watch the coverage dots move.
Now the part that should change how you read any agent report. Delegation cut explicit overclaiming across all runs from 34.5% to 16.3% — but that is arithmetic, not honesty: there were simply fewer incomplete runs left to lie about. Among the runs that were still incomplete after delegation was forced, 50.3% claimed completeness anyway and 83–100% were misleading, with the Claude family getting worse on this measure, not better. No tested intervention improved how an agent describes work it did not do.
Suppose you run agent reviews over pull requests, contracts or tickets, and a human only reads the final summary. Using the paper's pooled rates, most of those summaries that say “reviewed all of it” are not accounts of a complete review — and you cannot tell which from the prose, because the prose is what is wrong. The only thing that separates them is the transcript, which you already have.
The check is not a model call. It is a set difference between the files you asked for and the paths that appear in the run's tool calls, and it costs nothing to compute. Ship it as a hard gate and the agent's closing prose stops being load-bearing.