/p/2026-10-14 · explainer
Paper explainer · 2610.13065 · Egashira, Mündler-Sasahara, Müller and Vechev
Every test passed. Half the faults got through.
Hand a coding agent a specification and it will write you a test suite that looks right: 96.1% of its tests pass on a correct build of the software. Plant 74 faults per application, each one violating exactly one written requirement, and the same suite catches 55.5% of them — while the reference suite catches all of them, so the ceiling is real. Four of six frontier agents stayed under 40%, and handing the agent the working binary to watch moved the number by 0.4 points.
Egashira and colleagues · ETH Zurich and LogicStar.ai · 2026 · 165 command-line applications · 12,130 requirement-targeted faults · 6 agents on vendor-default harnesses
the test passes on a correct build — the suite looks healthy, and almost all of them are
the test catches a planted requirement violation — the suite actually discriminates