Two sentences broke the grader

Three papers on the instruments you trust to tell you whether a change helped: a grading prompt, a skill the agent never opens, and a reward that moves six points depending on which model scores it.

Today's pick

14 of 17

open-weight graders pushed out of usable accuracy by a two-sentence harsh-grader preamble

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

On a 570-student computer-vision exam the best model grader landed 1.64 marks out of 35 from the two-human average, where the two humans disagreed with each other by 2.61 — better than the people it would replace. Then the prompt turns out to be the experiment: a two-sentence harsh-teaching-assistant preamble pushed 14 of 17 open-weight models past the point of usefulness, three stopped awarding marks at all, and the sentence “never give partial credit” alone made two of three probed models refuse, while closed flagships shifted calibration but held. On a second exam the same preamble helped seven models — not from robustness but because their neutral prompt had been over-marking and the harsh wording cancelled that bias (the neutral bias predicts the direction, correlation −0.73). Sweep persona wording as a tuned parameter, judge it against a human-versus-human floor rather than an absolute target, and note that a light fine-tune on about 3,900 graded examples cut persona sensitivity to at most 0.32 marks.

open the explainer → abstract PDF

40 of 40 vs 0 of 40

audit runs in which the agent actually opened a skill, best configuration against three that never did

Demystifying Agent Skills for Smart Contract Auditing: Design, Effectiveness, Behavioral Impact

A skill — a folder of instructions, checklists and tool recipes a coding agent can load on demand — only does anything if the agent opens it. Eighty-three real security-audit skills collected from skill marketplaces and public repositories were run over 40 real audit tasks holding 120 known bugs, and whether they were read at all is the whole story: one setup loaded a skill in all 40 runs and found 22.8% more of the bugs, worth 43.2% more in bounty value, on 5.2% fewer tokens; another managed 13 of 40, a third 3 of 40, and three configurations never triggered a single skill across 40 runs each. Put a weaker model behind the same three harnesses and the effect turns negative everywhere (−5.9% to −9.4%), with the gap between harnesses inside 11% — the backend model decides, not the harness. Grep your traces for whether the skill was loaded before you credit it, and make its trigger conditions explicit rather than hoping the model notices them.

open the explainer → abstract PDF

4.92 / 6.09 / 10.63

points of reward gain for the same 64 reports, scored by three different model judge families

Evaluating Agent Skills for Version-Specific Plugin Migration: A Retrospective Study

The same 64 agent reports, re-scored by three different model judges, hand the same skill gains of 4.92, 6.09 and 10.63 points — so “the skill is worth about five points” is a claim about the judge as much as the skill. This retrospective traced all 328 pass/fail decisions behind one shipped plugin-upgrade skill back to what the target version actually requires, and found the rise from 93.83 to 98.75 was almost entirely one task: that task gained 42.5 points on its own, dropping it leaves +2.42 across the other fifteen, and eight of sixteen task pairs already scored 100 in both arms. Correcting one wrong decision — a path-containment check given full marks even though the literal parent directory slips through it — moved the confidence interval across zero. Report the ceiling effects and the per-task spread, not the mean, and re-score a sample with a judge from another model family before believing a small gain.

open the explainer → abstract PDF code