Agent models are usually scored one decision at a time: hand the model the correct conversation so far, ask for the next action, compare it against a reference. Fine-tuning four small models on real customer-support workflows raised that score for every one of them — 22 to 29 points on the text turns. Then the same models ran the same conversations end to end on their own output, and the best strict completion rate was 10.4% while two of the four never finished a single conversation. Under a holistic judge, no model completed a tool-requiring workflow at all, before or after fine-tuning.
The standard agent eval takes a recorded conversation, walks to turn k, gives the model everything that happened up to that point exactly as it really happened, and scores the one action it produces next. It is a clean protocol and it isolates decision quality well. It also repairs the world before every question: if the model got turn three wrong, turn four is still asked from a state where turn three went right.
Walk a support workflow with the repair switched on, then switch it off.
One wrong action does not cost one point in production. It changes what the next question even is. The paper’s closed-loop protocol replays the same conversations with a deterministic simulated user, so the only thing that differs is whether the assistant turns come from the transcript or from the model.
Supervised fine-tuning on this domain did what it is supposed to do. Judged turn by turn against the gold history, every model improved, and the text turns — what the assistant says to the customer — improved a lot. The tool turns improved too, by much less, and on the strict deterministic check of whether the function call exactly matched the reference, one model went backwards to zero.
Switch between what the model says and what the model calls.
That split is the first crack. Text turns rose 25.2 points on average; tool turns rose 5.9. A single headline “next-turn success” number blends the two, and the half that moves is the half that does not touch anything.
Same models, same conversations, one change: the assistant history is the model’s own. Seventy-seven conversations, a deterministic simulated user, and completion measured two ways — strict, meaning the model produced the reference trajectory, and holistic, meaning a judge read the whole thing and decided whether the customer’s problem was handled within policy.
Before fine-tuning, every model completed zero conversations. After fine-tuning, the two Qwen models completed three and eight of seventy-seven and the two Gemma models still completed none. Under the holistic judge the column is zeros all the way down, which tells you something the strict number does not: the eight conversations that matched the reference trajectory were still judged to have failed the customer or broken policy somewhere along the way.
Two mechanisms, and they multiply each other. The first is arithmetic: a workflow is a chain, and a chain of independent decisions succeeds at roughly the per-decision rate raised to the length of the chain. The second is that the gains were concentrated in prose while the actions barely moved — and it is the actions that the next turn depends on.
Set how good each action is and how many of them the workflow needs.
The independence assumption is generous in both directions — real errors correlate, and a good model recovers from some of them — but the shape is the point. It explains why a protocol that repairs the state between questions cannot predict the protocol that does not, and why chasing a per-turn number produces exactly the result the paper reports.
Most teams shipping an agent have one eval harness, and it is almost always the cheap one: replay a recorded conversation, score the next action. It is cheap because the state comes from the transcript, which is also precisely why it cannot see the failure that matters. The fix is not exotic; it is a second harness with a simulated user and no transcript to lean on.
Choose what your release gate measures and see what it lets through.
The cost difference between these gates is real: a closed-loop run needs a user simulator, a deterministic environment and roughly one conversation’s worth of tokens per sample, against one turn for the cheap version. Run the cheap one on every commit if you like. Do not let it decide what ships.