A model marking a 570-student computer-vision exam misses the two-grader average by 1.64 of 35 marks. The two humans miss each other by 2.61. Then a two-sentence harsh teaching assistant preamble goes in front of the same rubric and 14 of 17 open-weight models leave the usable band, three stop awarding marks at all. One sentence — never give partial credit — does it alone in two of three probed models. On a second exam from another course the same preamble helps seven models, because their neutral prompt had been over-marking and the harsh wording cancels the bias. A small adapter trained on about 3,900 real graded examples takes persona sensitivity down to 0.32 marks.
One long-form exam in a large course costs hundreds of grader-hours, and the people who can do it are scarce. So the question is not whether a model grades well in the abstract, it is whether it grades within the error the course already tolerates — the disagreement between the two humans who mark every script today.
On the computer-vision exam that floor is 2.61 marks out of 35: hand the same notebook to two qualified graders and their totals differ by that much on average. The best model configuration of the 171 tested came in at 1.64. It is genuinely tighter to the two-grader average than the graders are to each other, which is the result that makes anyone consider shipping this.
The paper then draws a line at 3.07× the floor — 8 marks on this exam — and calls anything past it out of the graded band: still producing numbers, no longer producing marks. Drag a grader across the scale and see where it lands.
The second exam, a machine-learning paper from a different course with 1,038 dual-graded students, is marked out of 65 and has a looser human floor of 5.13, so its band line sits at 15.7. Same rule, different scale — which is the point of having a floor at all rather than a fixed target.
Nothing about the rubric changes. Nothing about the notebooks changes. The only edit is the sentence the grader is introduced with. The neutral version reads “You are a strict but fair teaching assistant.” The harsh version reads “You are a HARSH teaching assistant. Award the MINIMUM defensible score for any task that is incomplete, buggy, or deviates from the rubric. Never give partial credit if the task does not run correctly.”
That is the whole intervention. Flip it and watch 17 open-weight models redistribute.
The named casualties give the scale of it. A 32-billion-parameter coding model sat at 7.49 marks under the neutral prompt — already marginal — and went to 20.29 under the persona. Two others landed at 25.05 and 26.04, which on a 35-mark exam means the grader and the humans are barely discussing the same script. The closed flagships of three vendors shifted their calibration under the same words and stayed inside the band, so this is not a law of language models; it is a property of the ones most teams can self-host.
The obvious explanation is that the model was told to be mean and got mean. The paper rules that out first. Swap the adjective — STRICT, RIGOROUS, FAIR — while keeping the two policy sentences word for word, and the error moves by between 0.94 and 2.15 marks across five models, in no consistent direction. Tone is noise.
The damage is in the two sentences that tell the grader how to withhold credit. Build the preamble yourself.
One sentence — never give partial credit if the task does not run correctly — is enough on its own to stop two of the three probed models from grading. It reads like a clarification. It is a policy.
Run the identical preamble on the second exam and seven of seventeen models get better. That is the finding most likely to mislead you, because it is not robustness. Those models’ neutral prompts had been over-marking — handing out more credit than the human pair — on 15 of the 17 models for that exam. A persona that subtracts credit cancels a bias that was already there. Across the 16 non-refusing pairs the size of the neutral bias predicts the size of the strict effect at a correlation of −0.73.
The cancellation is worth seeing as arithmetic, because it explains why a persona sweep on one dataset tells you nothing about the next. Set how much your neutral prompt over-marks, and how many marks the persona subtracts.
The fix is not a better prompt. It is a few thousand examples of the marking you already did. A light adapter — low-rank fine-tuning, the cheap kind you can run on one machine — trained once on the two exams’ pooled ~3,900 graded examples, then evaluated on held-out students from both.
Five small open models go from ordinary to at or better than a human grader’s agreement with the grader pair. Step between the exams.
The part that matters more than the accuracy is what happens to the persona. Sweep the same adapters across three harsh personas, on both exams, and the worst movement is 0.32 marks on the marks recipe and 0.62 on the breakdown recipe. The sentence that took a base model from usable to silent now barely registers. A grader that has seen your rubric applied to your students does not need to be told how mean to be.
You are probably not marking exams. You are probably running a rubric prompt over model output — scoring a support reply, ranking two answers, gating a release — and treating its number as ground truth. The measured graders below are the paper’s; the volume, the tolerance and the totals are yours, so they are labelled illustrative.
Tolerance is the honest question: how far can one score be wrong before the decision it feeds flips?
The cheap move is the one the paper ends on: keep a small set of double-scored items, report your judge against that pair rather than against an absolute target, and re-run the whole sweep whenever the rubric prompt is edited. A persona sentence is a deploy.