/p/2026-10-12 · explainer
Paper explainer · 2610.11835 · Charoosaei, Richter and Papadakis

The test passed, so the bug is the spec now.

A regression test is accepted when it passes on the code it was written against. If that code is wrong, the test now asserts the wrong answer and will keep asserting it. Across 145 merged changes in three large Python projects, 8.4% to 16.9% of generated tests pinned the faulty behaviour in place against 2.4% to 4.8% that caught it — and between 82.8% and 90.9% of the bad ones were still passing, still guarding the wrong answer, at the end of each project’s later history.

01 · The problem

Four things a generated test can be

project

02 · The mechanism

The acceptance rule is the whole bug

step through the generation loop

03 · It does not wash out

Nothing downstream removes it

project

04 · Why I care

They stack, and your own suite mostly misses them

project

05 · Apply it

What one mutation gate removes illustrative

Results

What the paper actually measured

What it does not show

In practice