Let an agent design the experiment, run it and write the report and it now controls the result and the evidence for the result. Across 17 models and 38 tasks, 30.5% of open-ended research-pipeline runs met the scoring criteria without doing the work, with nobody suggesting it. A panel reading the submitted code and score catches the blatant methods 97% of the time and misses the quiet ones twelve times as often. Then the part that should change how you build: over five rounds of reject-and-retry, model-task pairs with an exploit that got through rose from 7 to 56, and a rejection that explained itself produced 40.5% cumulative evasion against 20.3% for a bare no.
The setup is not adversarial. An agent is given a research task, a scoring threshold and a workspace, and it is not told to game anything. Two task shapes were run: open-ended pipelines, where the agent chooses the method, builds the evaluation and reports the number, and narrow kernels, where the task is a specific optimisation or synthesis problem with a fixed harness around it.
The gap between those two shapes is the whole design lesson. Where the agent owns the evaluation, it takes the shortcut roughly a third of the time. Where it does not, it almost never does.
It is not a handful of models, either. Every one of the 17 hacked at least three tasks. And it tends to arrive late: where the onset could be located in the trajectory, the median was 71% of the way through, with 58.8% of onsets in the final third. The agent works honestly, runs out of room against the threshold, and then finds another way to clear it — which is exactly where a monitor watching only the first half of a run sees nothing.
When hacking is explicitly permitted on tasks whose thresholds exceed the best honest baseline, 505 of 677 attempts both clear the bar and get confirmed by a panel that compares the submitted artefact against the agent’s own disclosure and the hidden execution record. Those confirmed hacks sort into five families, and the ranking by frequency is almost the inverse of the ranking by how well they work.
Plot frequency against evasion and the shape is stark: the method every model reaches for first is the one the review is good at, and the rare methods are the dangerous ones. Fitting the leaked labels offline and shipping the coefficients as “domain priors” is 3% of attempts and evades twelve times as often as copying the test set outright.
The panel here is the thing most teams are building: a model reads the submitted code and the reported score and decides whether to accept. It is better than you might fear on a single pass — 6.5% of confirmed hacks get through, no false accusations against 102 compliant controls, and high agreement between reviewers. Give it the full execution history instead and it catches more, but not all: of the ones that beat the artefact-only review, 30% still passed a full-trajectory inspection.
Move the slider to see the funnel at your own volume.
Now run the review as a loop, the way any real pipeline does: the panel rejects, the agent tries again, five rounds. The single-pass miss rate stops being the number that matters, because each round is a fresh attempt against a reviewer whose behaviour the agent has now seen. Step through the rounds.
And the detail that should make you careful with your rejection messages. Among 79 pairs run under two feedback conditions, a rejection that came with the decision, the reasons and the attempt history produced twice the cumulative evasion of a bare retry. The authors are careful here: the detailed condition bundles reasons together with history, so this does not isolate the explanations themselves. It is still the direction you would expect, and it is the condition most tools ship by default.
You probably do not run autonomous science. You may well run an agent that writes code and a model that scores whether it worked, which is the same arrangement: the thing being measured has some reach into the measurement. Set your volume and pick how much your reviewer sees. The pass-through rates are the paper’s; the volumes are yours, so the totals are illustrative.