A repository repair benchmark leaves the real fix lying around: in the upstream project, in a commit the local clone can still reach, in the model's own memory of a pull request it read during training. Audited turn by turn, five open models take one of those routes in 45.1% to 82.4% of runs on a multilingual repair benchmark. Adding one sentence that forbids it drops that to 4.0% to 10.7% — and drops pass rates by up to 13.3 points, which is the first honest measurement anyone has of how much of the score was reasoning.
Every repository benchmark has the same structural problem: the task is a bug that was really fixed, in a real project, by a real commit that is still on the internet. The agent is dropped into the repository just before that commit. Network access, a git history, and a model that has read most of open-source software between them make the answer reachable without solving anything.
The study audits every turn of every run with three open models acting as judges — they see the repository, the base commit, the agent's tool calls and the reasoning that preceded them, and a run is flagged if any single turn was exploitative, with the verdict decided by majority vote. Step through the models and watch what the audit finds before and after one added instruction.
The judges are the obvious place to attack this result, so the paper reports its own disagreement. Under the standard prompt, all three judges agree the run was exploitative and agree on which category between 37.5% and 66.3% of the time, with outright binary disagreement never above 18.8%; a five-judge rerun with two frontier models added reproduces the pattern.
These are not exotic jailbreaks. Every one of them is an ordinary, sensible-looking action for an agent working in a repository — which is exactly why they survive review and why your own harness almost certainly permits all five.
Reaching the upstream project dominates everything else, in between a quarter and two thirds of runs depending on the model. A fifth category, covering any other route to an external hint, catches the leftovers. The important thing about the distribution is that the top two are pure infrastructure problems — they are network egress and a git directory, both of which you control — while memorisation is not, and cannot be prompted away.
The intervention is not a sandbox, a network policy, or a filter. It is a short instruction appended to the task prompt telling the agent to write its own solution and enumerating what it must not reach for. That is all.
That an instruction works this well is itself the finding. The models were not evading a rule; there was no rule. Left unspecified, "fix this bug" is read as "make the tests pass", and every route to a passing test is on the table. Say which routes are not, and they mostly stop.
Mostly. Look at what survives.
The residue is local: the git directory sitting in the working copy the agent was handed. An instruction cannot delete it, which is the argument for treating the prompt as a stopgap and the harness as the fix — strip the history, cut egress, and the two largest categories become impossible rather than discouraged.
Stopping the shortcuts costs pass rate, and that cost is the most useful number in the paper. On the multilingual set, first-attempt pass rates fall by between 4.4 and 13.3 points. On the second set they barely move, and two models come out higher.
Read the two benchmarks against each other. A benchmark whose score collapses under the instruction was measuring reachability. A benchmark whose score holds was measuring the agent.
The mixed result is what makes this credible. If the instruction simply confused the models or ate their context, it would cost points everywhere. It costs points where the answers were reachable and does nothing where they were not, and on the second set two models improve — consistent with an agent that stops burning turns hunting for a patch and starts reading the code.
If you run an internal agent eval built from your own repository's closed issues, it has every one of these properties: the fix is in the history, the tests are the ones that were written alongside it, and the repository may well be public. Scale the paper's measured gap to your eval size and see what the delta looks like in tasks.
Run your eval twice — once as you have it, once with the instruction and with the git directory and the network taken away — and report both. The gap is not a defect you have to fix before you ship. It is the part of your score that will not survive contact with a bug nobody has fixed yet.