A panel of LLM judges is treated as a panel of independent opinions, and every significance test run on its votes assumes exactly that. But judges built and tuned the same way are wrong on the same items. Across a bank of ten open-weight judges the errors correlated at 0.21, leaving the panel with about as much independent evidence as 3.5 judges; three frontier judges from three different providers correlated at 0.56 and were worth 1.4. Pool the votes as if they were independent and the variance is understated 2.85× — enough to flip the verdict in 28% of comparisons.
Stand up a jury of judge models, take the majority, and the number you report has a sample size attached to it — ten judges, a thousand votes, a comfortable-looking confidence interval. That arithmetic is only right if the judges are wrong independently. When their mistakes line up, the panel is quietly smaller than its headcount, and there is a standard expression for how much smaller: divide the number of judges by one plus their average pairwise error correlation times one less than the count.
Move the two things that decide it and watch the panel shrink.
The shape is the point. At a correlation of zero the two lines sit on top of each other and a panel really is worth its headcount. At the measured 0.21, the curve flattens almost immediately: the eleventh judge on a ten-judge panel is worth a rounding error, and no amount of buying more of them gets you past roughly one over the correlation. You cannot spend your way out of a shared blind spot.
The authors scored five open-weight models under two prompting styles — ten logical judges — on a stratified slice of a public preference benchmark, recorded which items each got wrong, and measured how far those error patterns overlapped. Then they did it again with frontier judges, within one provider and across three. The result runs the opposite way to intuition: the better and more similar the judges, the more their mistakes coincide, and the less a panel of them is worth.
Step through the banks the paper measured.
Three frontier judges drawn from three different providers correlated at 0.56 — more tightly than three models from a single provider, which managed 0.42. Different company, different training run, same blind spot.
What that looks like item by item is worth seeing, because it is the thing a majority vote cannot detect. Below, each column is one item you are judging and each row is a judge; a filled cell is a judge getting that item wrong. Raise the correlation and the errors stop scattering and start stacking into columns — items the whole panel misses, where the majority is confidently, unanimously wrong.
Here is where it stops being a statistics curiosity and starts changing decisions. The usual way to test whether system A beat system B on a judged panel is to pool every vote — ten judges times a hundred items is a thousand observations — and run the test on that. The honest alternative treats each item as the single observation it is, because the ten votes on one item are not ten looks at the question. The paper ran both on identical vote sets.
Set the panel up as you would run it, then switch how the votes are counted.
Across resampled hundred-item panels the two tests disagreed constantly. On the focus panels the pooled test called a significant winner 79% of the time and the item-level test 52% of the time; the gap is the share of runs where you would have shipped on a result that was not there. One panel in the paper produced z = 2.03 pooled and z = 1.52 per item, from the same thousand votes.
Note which columns are safe. Verifiable tasks — arithmetic, precise instruction following — flip in under 2% of comparisons, because there judges mostly agree with the answer key rather than with each other. The damage is concentrated in exactly the judgements you cannot check automatically, which is the reason you hired a panel of judges in the first place.
The average correlation tells you how much evidence you have lost. It does not tell you what to do about it, because the same average can come from two different shapes of failure. Either most of the panel goes down together on certain items, or a subgroup of judges shares a specific weakness — a position bias, say — and the rest are fine. The authors induced both shapes deliberately and tested three aggregation rules against each.
Pick a failure shape and see which rule survives it.
The asymmetry matters more than the winner. Matching the rule to a vulnerable subgroup bought 4.9 points of precision; applying the correlation-downweighting rule to the wrong shape cost 3.3 points. And the authors' attempt to learn the choice automatically did not transfer — a router that identified the shape correctly 85% of the time delivered no reliable precision gain, and a selector trained on one benchmark lost 0.74 points when moved to another. The choice has to be made on your own labelled examples, not inherited.
The concrete version: you changed a prompt, ran both variants over a panel of items, had a jury of judge models score them, and the dashboard says the new one wins. Set the panel below to whatever you actually run, and give it an honest guess at how alike your judges are — same provider and same prompt template is high, a deliberately mixed bank is lower.
The cheapest fix is not a bigger jury. It is a hundred items you have labelled yourself, scored once by every judge you use, which gives you each judge's accuracy and, more usefully, the map of which judges go wrong together. That map is what turns a panel verdict back into evidence — and it is the same hundred items you can reuse to choose the voting rule before you apply it to anything that matters.