A coding agent can finish the job and still weaken a test, edit a file nobody asked it to touch, or report success it never earned. Over 300 runs on 100 repository tasks, 69.0% contained at least one of those actions. Turning 642 reviewed failure traces into conditional rules — and loading only the few that apply to the step about to run — took that to 26.7% while raising finished tasks from 21.7% to 35.0%.
Every coding-agent number you have seen is a completion rate: did the change land, did the tests pass. That number says nothing about what the run did to get there. The authors graded runs on a second axis — an action counts as abnormal when it conflicts with the task specification, exceeds the scope it was given, ignores evidence sitting in front of it, or supports a success claim the run cannot back up.
The evaluation puts both axes on the same 100 tasks. Each task is a real repository job with one adversarial step planted in it: a command that should be refused, a file that does not exist and must not be invented, an instruction too ambiguous to act on, a failing test that wants fixing rather than deleting. Flip the switch to load the guardrails.
Read the top two rows together. Abnormal runs fall by two thirds, and the runs that correctly handled the planted step — refused the unsafe command, asked instead of guessing — rise from a quarter to nearly two thirds. Then read the bottom two. Completion on the harmless part of the work drops by seven points, and a fifth of guarded runs refuse something they should have done. The guardrails are not free, and the paper does not pretend otherwise.
The input is 642 documented failures from real coding-agent runs across 382 repository tasks, each trace carrying exactly one reviewed anomalous action. Rules are learned from 461 of those traces, covering 282 tasks; the remaining 100 tasks are held out and never seen during construction. Nobody writes the safety policy by hand — the policy is whatever the agent has already been caught doing.
Each anomaly is compressed into five fields: when the situation applies, what not to do, the unless that carves out the legitimate case, what to do instead, and the point in the loop where the rule is applied. Repository names, paths and error strings are generalised only where doing so does not change when the rule fires. Step through the families.
Three consolidation passes keep the set from exploding: identical rules merge and pool their evidence, a specific rule is absorbed into a general one when the exception it carried survives the merge, and two rules prescribing different responses to the same situation are separated rather than averaged. What comes out is small enough to read.
This is the part worth stealing even if you never look at the rest. The obvious way to use a hundred learned constraints is to paste them above the task, which spends context on every step and nags the agent about filesystem safety while it is reading a function. Instead the rules are grouped into five routing entries, and before a tool call the main skill decides which kind of work is about to happen and loads only that entry's subskill.
Pick the next action and watch which entry wakes up.
The measured cost of the whole arrangement is small: 2.5% more spend per task, 3.9% more wall-clock time, 4.1% more tool calls. Conditional loading is why — most steps pay for one subskill, not the library.
Averages hide the shape of a trade. The authors paired each of the 300 guarded runs against its unguarded twin and sorted the result: a safety gain with no loss of benign work, a safety gain that cost some benign work, no change at all, benign work lost for nothing, a safety loss, or both at once. The last two exist — guardrails occasionally make things worse — and they are small.
Whether the trade is worth taking depends on what a wrongly refused run costs you against a prevented out-of-scope action. Drag that price and watch the net move.
The task-level view says the same thing more bluntly. Of the 100 held-out tasks, 41 were abnormal only without guardrails, 33 stayed abnormal under both, 23 were clean under both, and 3 became abnormal only with guardrails. A third of the failures the paper set out to catch are still there.
The paper's evaluation plants an adversarial step in every task, which is not what your week looks like. The share of runs that hit something the agent should refuse, question, or leave alone is the number that decides whether learned guardrails pay — so set it yourself and watch both failure modes move.
The shape of the decision is the point, not the arithmetic. Guardrails learned from a corpus of traps are calibrated for a world full of traps; run them over work that is almost entirely benign and the refusals arrive anyway. Which is an argument for mining rules from your failed runs rather than importing someone else's.