/p/2026-10-14 · explainer
Paper explainer · 2610.13303 · Han and Kim

They agreed with each other, not with you.

When a panel of judges splits on a score, the usual reading is that the rubric is unclear. So rank a rubric's clauses by how often the judges disagree, rewrite the contested ones, and measure both numbers. Judge-to-judge agreement rose 0.069 — and 0.074 above a control that edited the same number of untargeted clauses. Agreement with the expert ratings moved −0.007, and that untargeted control finished closest to the humans.

01 · The problem

Disagreement looks like a rubric bug

which benchmark's rubric

02 · The mechanism

Rank the clauses, rewrite the top k

pick a dimension — its clauses, in the paper's own order

03 · The result

Two numbers, and only one of them moved

which dataset

04 · Why I care

Consensus was the objective, and it was the wrong one

rubric the judges were reading

05 · Apply it

What the same experiment costs on your rubric illustrative

Results

What the paper actually measured

What it does not show

In practice