The right tools in the wrong order

Three papers on the parts of an agent loop nobody writes down: which tools it can see before it moves, whether it should be editing at all, and when it is allowed to stop.

Today's pick

74% → 90%

multi-step tool tasks the agent completed, stock 32-tool shortlist versus a route-ordered one

The Menu Is an Execution Prior: State-Path Tool Menus for Online Agents

An agent with thousands of tools available only ever sees a shortlist, and that shortlist is normally built by matching the request against tool descriptions — which reliably surfaces the tool that produces the answer while burying the dull tools that produce its inputs. Ask for a receipt to be emailed and the send-receipt tool ranks first, but it cannot run until an order lookup, a receipt build and an address lookup have fed it, so the run stalls with the right tools present in the wrong order. This work builds the shortlist from routes instead — which tools can run from the current request state, whose outputs satisfy whose inputs, which orders recurred in recorded runs — then puts producers ahead of consumers, taking completed multi-step tasks from 74% to 90% with the agent itself unchanged. A 32-tool routed shortlist covered more complete tool chains than the stock 128-tool list, so the lever is ordering and coverage, not a bigger context window.

open the explainer → abstract PDF code

29% vs 6%

chance one review round breaks a passing program versus repairing a failing one

If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs

Point a model at code and ask whether it has bugs, with no failing test to anchor the question, and it keeps finding them. Across two models and 800 competitive-programming submissions, a single review round was far more likely to break a program that already passed than to fix one that did not — 29% against 6% for the stronger model writing search-and-replace edits. Run that loop blindly and it settles into a pseudo-fix cycle, adding and removing the same change forever; the two rates compound, and only about one program in six is still correct once the loop settles. Probing the activations turned up a single direction that controls how eager the model is to edit at all: suppress it and the loop stops dead, preserving every correct program and fixing none of the broken ones. If you run an automated review or repair pass, make a failing test the precondition for an edit, because the loop has no stopping condition of its own.

open the explainer → abstract PDF

1 stop in 9

early stops landing on an answer the model itself later abandoned, at a 32% token saving

Stable Answers, Unfinished Reasoning: Why Self-Consensus Is Not a Safe Early-Exit Signal

A cheap way to cut reasoning cost is to keep asking a half-finished chain of thought what its answer is so far and stop once several checks in a row agree. Swept across 3,520 versions of that rule on frozen traces from four models and three competition-maths sets, not one met any of three cost-and-accuracy targets fixed in advance: at the setting still saving 32% of tokens, one stop in nine fires on an answer the model itself later abandons, and those stops mostly cut off a correction it was about to make — 155 corrections lost against 57 answers rescued. Widening the agreement window does not rescue it, because the share of bad stops flattens near 7% while the saving falls to 8%: agreement shows only that an answer is stable under a fixed question, not that the reasoning has finished. Reading the model's confidence at a natural break in the reasoning, swept through the same pipeline, cleared all three targets — so gate a reasoning budget on that instead.

open the explainer → abstract PDF