The tests passed, the patch was wrong

Three papers on the distance between a green check and working software: one replaces the test suite with a proof, one finds that most agent failures happen after the tool call, and one shows a fine-tuning run that lifted every per-turn score and completed no workflows.

Today's pick

85% → 58%

issues counted as resolved before and after a machine-checked audit of patches that passed every hidden test

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

If a model's patch passes every test in the repository, most teams call the issue fixed, and on 500 real issues that verdict was wrong often enough to matter. The authors rebuilt those issues so correctness is decided by a machine-checked proof against a written specification instead of by the test suite, and the resolved rate fell from 85.0% to 58.2% for the stronger model and from 81.2% to 33.4% for the other, on patches that passed everything and still admit an input where they differ from the reference fix. Handing over a correct specification pushed resolution back to 94-96%, but models writing their own passed the audit only 46% to 72% of the time, and the dominant failure was a specification that pins down part of the required behaviour and leaves the rest free. You are not going to run a proof assistant over your agent's pull requests; the move you can copy is to treat your acceptance checks as the specification and write the half you are missing.

open the explainer → abstract PDF

75.7%

share of agent failures that came from runs which had already invoked a tool

What Stops a Small Language Model From Driving a Database Agent

When a small open-weight model fails at agent work, the usual explanation is that it cannot reason well enough. Eleven days of driving a real database client's agent mode with 39 locally served models produced 2,100 attributable losses, and 75.7% came from runs that had already got a tool call out; the largest class used its tools, did not time out, and never returned a deliverable. The ledger stored refusal codes and never the arguments the model sent, so those failures were indistinguishable from incapacity — capturing the arguments exposed five server defects, one of which demanded a field on one tool, forbade it on the sibling that composed it, then failed the run for its absence. Fixing them moved six models by 6 to 21 cells out of 30 while touching no model, prompt or sampling setting, so log what your agent actually sent on the calls your server rejected before you blame the model.

open the explainer → abstract PDF code

10.4%

best end-to-end workflow completion rate, after fine-tuning raised every model's next-turn score by 22 to 29 points

When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success

Agent models are usually scored one decision at a time: hand over the correct history, ask for the next action, compare it against a reference. Fine-tuning four small models on customer-support workflows raised that score for all of them, text turns by 22.6 to 29.0 points, and then the same models ran those conversations end to end on their own output, where the best strict completion rate was 10.4% and two of the four completed nothing; a judge reading whole conversations passed no tool-requiring workflow anywhere in the study. Gold-history scoring repairs the state before every question, so it never charges the model for the mistake it made two turns ago, and the gains were in the prose anyway: judged tool-turn success rose 5.9 points on average against 25.2 for text. Report text quality, action correctness, tool execution and completion separately, and keep the per-turn eval for fast feedback rather than as a release gate.

open the explainer → abstract PDF