The checks were there. The contract wasn't.

Three papers on success signals that do not hold: a validator that checks the wrong thing, an eval rubric that quietly rewrites its own standard, and a coding agent that accepts a predecessor's claim of done as the definition of the task.

Today's pick

5.3%

of call sites that enforce the whole contract their job requires, though 82% do check something around the call

When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems

Every place your code calls a model is a boundary with an unwritten agreement: what must hold before the call, what the request has to say, and which replies you accept before something acts on one. The authors wrote down the agreement each of 496 call sites in 40 open-source agent systems needs — deriving it from the job, without reading the code — then compared it against what the code enforces: 82% of call sites do check something before and after the call, yet only 5.3% cover their whole agreement, and two thirds of the constraints belonging in the request are left as instructions to the model rather than verified in code. Placement is what bites, because 48% of model outputs feed control flow or execution and 73% of those never reach a person. Ten recurring shapes explain 99% of the gaps, which makes them a review checklist worth running over your own call sites; start with the reply that fails open, where an invalid answer silently becomes the permissive default.

open the explainer → abstract PDF

2% vs 100%

share of known-good outputs that pass, under a generated checklist versus a hand-written one with the same judge

Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

When you grade model output with another model, the argument is usually about which judge to use. Varying the pieces separately shows the checklist is nearly all of it: holding one judge fixed, six hand-written checks passed 96-100% of known-good outputs while twelve auto-generated ones passed 2-8%, and swapping the judge model or rewording the checks moved results by a few points. Letting a model improve its own rubric made it worse every time — over four rounds it added requirements the task never stated and deleted checks that were doing unique work: of 186 deleted checks, 179 had caught a real flaw and 78 were the only check catching it. Version your rubrics like code, and if you let a model edit one, freeze the checks you derived from labelled examples and block any deletion that removes the sole catcher of a known failure.

open the explainer → abstract PDF

20.6-53.5 pts

drop in success rate when a coding agent starts from a half-finished attempt instead of a clean repository

SWE-Residual: Evaluating Starting-State Robustness of Coding Agents

Benchmarks start a coding agent from a clean checkout; production hands it a retry, a resumed session, or another agent's half-finished branch. This work rebuilds 198 such starting points — each the state an earlier agent left after declaring the task done while hidden tests still failed — and gives a fresh agent only the original request and that state, with no hint the previous attempt failed. All five frontier agents fell 20.6 to 53.5 points below their clean-start scores, and the order flipped: the agent 17.8 points ahead on clean starts was 15.2 points behind on residual ones. The cause they name is acceptance bias — agents read the inherited work as the task's scope, keeping 86-98% of the predecessor's lines and ending 59-88% of their failures with exactly the tests that were already failing, having opened the files the real fix needed and changed nothing. If your product resumes sessions or hands work between agents, make the first step re-derive scope from the original request rather than from the diff it inherited.

open the explainer → abstract PDF