/p/2026-10-13 · explainer
Paper explainer · 2610.13563 · Peng, Hu, Niu, Li and Xia

Both patches passed. One did twice the work.

Issue-resolution benchmarks score a patch on whether the test turns green. Measured on counted hardware instructions, 42.0% of 262 real issues contain a statistically significant performance difference between two correct patches, the models’ runtime ranking is nearly the reverse of their resolution-rate ranking, and in the paper’s worked case three patches that all pass the same test differ by 2.1×.

01 · The problem

Correct is not one thing

which benchmark split

02 · The mechanism

Three ways to pass the same test

pick a repair strategy

03 · The method

The test you already have is the benchmark illustrative

04 · Why I care

The ranking you trust is the wrong one

05 · Apply it

What one performance gate catches illustrative

Results

What the paper actually measured

What it does not show

In practice