A repair agent is accepted when its patch parses and the tests go green. That is exactly the bar a security fix can clear while changing nothing that matters. Across 1,030 repair runs from seven agent frameworks on two vulnerability datasets, 170 patches cleared it and were still unsafe — roughly half left the dangerous call in place under a cosmetic edit, and 30.6% introduced a hole the original code did not have. Every one of them would have been scored a successful repair by a standard benchmark.
Start with the study's worked case. The task is a command-injection repair: the original code builds a shell command out of external input and runs it. The agent's patch adds a length check on the input and renames the surrounding variables. The shell execution call is untouched.
Now run it past four levels of checking. The first two are what an agent benchmark actually applies. Step through them.
The length check is the tell. It is a real edit, in the right file, addressing something plausible about untrusted input — and it does nothing at all about the injection, because sixty-four characters is plenty of room for a semicolon. The tests never noticed, because the tests that shipped with the task use benign inputs. That is the definition the paper works with: a silent failure is a patch that passes syntactic and functional verification and still retains or introduces a vulnerability.
Out of 1,030 valid runs, 252 patches were flagged as candidates and 170 were confirmed by manual review across three rounds of coding. That is roughly one accepted repair in six.
The 170 confirmed cases sort into three categories, and the split matters because each one implies a different check. Pick a category.
Underneath those three sit ten fine-grained codes. Two of them account for nearly half of everything: insecure API preserved — the dangerous call carried over from the input with no secure substitute — and introduced new vulnerability, where the repair itself is the origin. Toggle the view between share and count.
Ten of the 170 are hardcoded secrets — a signing key, a token or a password written into the source as a string literal, by the agent, during a security repair. That one is worth singling out because it is the cheapest of all of these to catch: a secret scanner on the diff costs nothing and would have caught every one.
Seven frameworks were run, four of them multi-agent — an assembly line, a hierarchical workflow, a dynamic conversation and a test-driven loop — and three single-agent, running a reasoning-and-acting loop or an edit-and-feedback loop. They fail in measurably different proportions. Switch between them.
Omission is flat — within four points of each other — so the base failure is the model's, not the architecture's. What moves is the other two. A multi-agent system generates more artifacts, and every artifact is another chance to introduce something; a single agent grinding on one file is more likely to build a defence that is bypassable than to invent a new hole.
The frameworks are the ones you would recognise, and the study notes that similar insecure solutions recurred across them, which points at shared model, prompt or task influences rather than at any one framework's design.
The obvious mitigation is the one four of these frameworks already ship: put a reviewer role downstream of the writer. In the multi-agent cases the study examined, the reviewer rarely flagged a security deficiency before it reached the final output. Two reasons, both structural.
The study's confirmed rate is 170 silent failures in 1,030 valid runs, on Python, on security-focused tasks, with one small hosted model behind every framework. Your rate is not that rate. But the arithmetic is worth doing once, because the quantity it produces is "changes that merged with a hole in them, and nothing in the pipeline said so". Set your volume and choose what your merge gate actually runs.
The point of the ladder is that the two cheapest gates on it are the two nobody runs on agent output. A secret scanner is a regex pass. A reachability check on the diff is a linter. Neither of them needs a model, a benchmark or a security team, and between them they cover the largest single code in the taxonomy and the most embarrassing one.