A shipped skill that packages version-specific migration knowledge for a coding agent takes the recorded reward from 93.83 to 98.75 — a gain of 4.92 points. Trace all 328 pass/fail decisions back to what the target version actually requires and the gain is almost entirely one task: it alone gains 42.5 points, dropping it leaves 2.42 across the other fifteen, and eight of sixteen task pairs already score 100 in both arms. Correcting one wrong decision — a path check given full credit although the literal parent directory passes it — moves the interval across zero. Re-score the same 64 reports with judges from two other model families and the same skill is worth 10.63 and 6.09.
The skill is an ordinary, useful thing: a plugin framework ships often, each release renames exports or moves build artefacts or changes how sessions are represented, and somebody writes down what the upgrade needs so the coding agent stops guessing. Sixteen diagnostic tasks, two attempts per arm, 64 reports. Mean recorded reward goes from 93.83 to 98.75.
Ninety-three point eight three is the number worth staring at. The arm without the skill was already close to full marks, which means there are only about six points on the table before you start. Take the top task out.
One task carries more than half the total. Eight carry nothing, because both arms already scored full marks and a rubric cannot record an improvement above its own ceiling. The seven that remain share the rest. The mean is doing a lot of work here that the distribution does not support.
The reward is not a judgement about the migration. It is a weighted sum over a checklist. Each criterion carries a weight; a pass takes the full weight, a partial takes half, a fail or a missing answer takes nothing. Three hundred and twenty-eight of those decisions produced the 64 scores above.
Which means the score moves in discrete jumps, and on a report already near the ceiling one flipped criterion is most of a task’s gain. Move the criteria and see.
Every one of those decisions belongs to a contract domain — whether the advice applies to the target release, whether an interface exists and who owns the data behind it, whether things happen in the right order, whether a boundary is actually enforced, what a failure is blamed on, and what evidence is offered. Tracing them by domain is what turns a score back into a claim you can check.
One with-skill answer proposes a containment check: a path is inside the base directory if it is not absolute and does not begin with two dots followed by a separator. That sounds like the normal defence. It is not quite one, and the judge awarded full credit for it anyway.
Try paths against it. The left verdict is the predicate as written; the right is what containment actually requires.
The exact string .. is not absolute and does not start with two dots followed by a separator, so it passes. It denotes the parent directory. The rubric asked for containment, the answer did not provide it, and the criterion was marked pass.
A second case runs the other way. A task’s rubric required that timers not keep a still-mounted host alive; the answer tore the host down explicitly instead. An executable probe confirms that the teardown works, but it is not what the rubric asked for, so the decision stands as written — a reminder that a rubric can be wrong in the direction of harshness too.
Put the corrections back and the estimate survives. Its evidence does not.
Nothing about the agent changes here. The same 64 reports, written once, are handed to judges from two other model families under the same rubric prompt, with no arm labels and no prior scores. If the reward were a property of the work, the three would agree on what the skill was worth.
Agreement of 91.8% and 95.7% on individual decisions, and the headline they produce differs by a factor of two. High agreement on a checklist is compatible with a very different answer to the question you actually asked.
There is one more thing worth knowing about the original judge: it and the agent being judged come from the same model family. Agreement between them does not validate the rubric, and the paper says so plainly rather than leaning on it.
Two properties are enough to produce a confident-looking gain out of almost nothing: most of your tasks already score full marks, and one of them has room to move. Set both. The tasks, the ceiling share and the outlier are yours, so the totals are labelled illustrative.
The paper’s own remedy is unglamorous and cheap: trace a sample of decisions back to the contract they claim to check, and re-score the whole set with a judge that does not share a model family with the thing being judged. Both of those moved the answer here more than the skill did.