Measuring social bias by having one model judge another's answers assumes the text being judged is clean. Real user writing is not: it carries typos, dropped punctuation, informal spelling and clause-final fillers. Applying five kinds of surface noise at five intensities to 3,822 answers and comparing each judge against its own verdict on the clean text, the damage runs almost entirely one way. At realistic corruption the weakest judge turned 240 neutral verdicts into biased ones and only 2 biased verdicts into neutral — a 120× imbalance. Stack the noise types and it becomes 288 against zero. Robust judges narrow the gap; none of the four reverses it.
The experiment is deliberately unglamorous. Take an answer, corrupt its surface without touching its meaning, and ask the judge again. Five corruptions, each applied at five strengths from barely-there to every-eligible-token. At the middling strength the authors call realistic, the share of tokens actually altered runs from 16% for stripped punctuation to 47% when three corruptions are stacked.
Pick a corruption and turn it up. The sentence is ours; the corruption definitions are the paper's.
The judging task itself is a three-way choice. The judge sees a context, a question, two options naming different groups, and the answer text, and returns one label: option A, option B, or neither. "Neither" is the neutral verdict — the answer does not favour either group — and it is the verdict that turns out to be fragile. Crucially, the comparison is against each judge's own verdict on the clean version of the same answer, not against a human gold label. That makes this a measurement of stability, not of correctness: a judge could be wrong on the clean text and still count as stable.
If surface noise merely confused the judges, verdicts would scatter in both directions: some neutral answers would be called biased, and some biased answers would be called neutral, in roughly equal numbers. That is not what happens. Pick a judge.
The ratio falls by two orders of magnitude as judges get stronger — 120 to one on the weakest, 1.3 to one on the strongest — and it never crosses over. Pooled across all hundred judge-and-condition combinations, erasure exceeds fabrication in only eight, and not one of those eight is statistically significant. The authors' reading is worth quoting in substance: robustness does not flip the direction of the distortion, it removes the distortion. There is no judge for which noise makes bias look smaller.
Two things behave counter-intuitively here. The first is stacking: combining informal spelling, typos and stripped punctuation produces more fabricated bias than typos alone, by between 1.2 and 1.6 times on the three open judges. Switch the corruption and watch every bar move together.
The second is intensity, and it is the finding most likely to catch a practitioner out. You would expect heavier corruption to mean a worse measurement, and in absolute terms it does — fabrications rise from 240 to 403 as typo noise goes from realistic to maximal. But erasures rise faster, so the ratio falls from 120 to 50. The distortion is at its purest at the noise level you would actually see in production, because at mild corruption the answers are still legible enough that nothing gets read as neutral by accident.
Fabricated bias is not spread evenly across the categories being measured. Under the stacked corruption at realistic strength, religion and disability draw roughly two and a half times as many fabrications as gender does. Pick a category.
The ranking holds across noise types and at full intensity, which is what makes it usable: it is a property of the categories rather than of one corruption. The practical consequence is uncomfortable. If your fairness report is computed on real user text and shows religion and disability as your worst categories, some unknown share of that ordering may be the noise in the writing rather than the behaviour of your model — and gender, the category most often reported on, is the one least inflated, so cross-category comparisons are the most distorted thing on the page.
Take a bias rate you would report, decide how much of the text it was computed over is messy user writing rather than clean synthetic prompts, and pick how robust your judge is. The per-judge flip behaviour is the paper's; projecting it onto your own corpus is arithmetic and labelled illustrative.
There are two moves, and they cost almost nothing. Normalise before you judge — spell-correct, restore casing and punctuation — and run the judge on the cleaned text, keeping the original only for the record. Then measure your own asymmetry: take a sample of your eval set, corrupt it deliberately, and count how many verdicts move in each direction. That number is a property of your judge, it varies by an order of magnitude between models, and it tells you how much of your bias report to believe.