Two coding agents sit one rank apart and you read that as one being better. Auditing 254 published submissions without rerunning a model: the leading two each resolve 396 of 500 problems, the top ten share 285 solutions and 51 failures, and a paired test separates none of the 29 adjacent pairs in the top thirty. What does move the number is the harness around the model — one model across nine scaffolds spans 29.8 points, against an 8.8-point spread for the whole top thirty.
A score is a count of solved problems, and two counts being close tells you nothing about which problems. The audit works from published per-problem verdicts, so it can ask the question the scores cannot: of 500 problems, how many actually distinguish one top system from another?
Switch between the leading pair and the leading ten. Grey is a problem they all solve, violet a problem they all fail, green a problem where they disagree — the only part of the benchmark doing any ranking.
The union is the other way to see it. Combine the top two and you resolve 414 problems — 18 more than either alone. Combine all ten and you get 449. Ten systems, built by different teams, and their collective reach is 35 problems beyond what the best pair already covers.
Overlap alone is not surprising — two systems with high scores must overlap a lot. The audit measures the excess: how much more of the weaker system's solution set the stronger one covers than their scores alone would predict, if they were succeeding independently.
For two systems A and B, the coverage is the share of B's solutions that A also gets. If success were independent, that share would just be A's score rate. Step through the score bands and watch the observed coverage pull away from it.
Read the direction of travel: the excess is largest in the middle of the table and smallest at the top, because at the top there is barely any room left to be nested in. These systems are not carving up the problem space differently. They are climbing the same ladder, and the ones behind are strictly behind.
Every leaderboard row is a model inside a scaffold: the retrieval, the file navigation, the edit format, the retry policy, the test loop. The audit groups submissions by model and measures how far apart the same model's own submissions land.
Pick a model. The violet band behind each bar is the entire spread of the top thirty submissions — the range everyone argues about.
The honest caveat the authors attach: this is observational. A scaffold that scores well may also be the one its team optimised hardest, so the spread is not a clean causal estimate of what a harness is worth. What it does establish is that quoting a score as a property of the model, with no mention of what was wrapped around it, is not a meaningful claim.
With per-problem verdicts you can compare two systems properly: ignore the problems they both solved and both failed, and ask whether the ones they disagree on split evenly enough to be chance. That is an exact paired test, and it is the right one — the systems are graded on the same problems, not on independent samples.
Applied to each adjacent pair in the top thirty, split by split:
The split with four and a half times as many problems separates 14 of 23 adjacent pairs; the popular 500-problem split separates none of 29. That is the whole story about resolution: the benchmark everyone quotes is too small for the differences that are left. Pricing the shortfall, the median adjacent pair would need about 52 times its current problem count — roughly 26,000 problems of the same character — before its gap became statistically real.
One caution the paper is careful about: failing to separate two systems is not evidence they are equal. It means the data cannot tell, which is a different and more honest claim than a tie.
The practical consequence is not "benchmarks are broken". It is that a gap has a minimum size before a given number of problems can see it, and most internal eval sets are far smaller than the one that already cannot separate the top thirty.
Drag your eval size. The curve is the smallest true difference you could expect to detect, scaled from the paper's own power calculation.
Which leaves a better procedure than reading the table. Pick the comparison set yourself — problems from repositories that look like yours — and report the score on that set, with the scaffold named. Where two candidates come out within a couple of points, stop treating the gap as a result and choose on the things that are not noise: what the harness does on failure, what it costs per resolved problem, whether you can read its trace when it goes wrong.