Train a tool-using agent with reinforcement learning and it learns which prompts go with which tool. "Looks like" is cheaper to learn than "needs". Dropping inert markers — a fake web-index tag, an answer-format instruction — into the search half of a 1,000-example training set left the agent calling web search on maths questions up to 39.2 points more often when the marker was present than on the identical question without it. A judge that scores each call for necessity took that to 0.0.
The setup is deliberately small and clean. A thousand training examples: five hundred factual questions that a web search helps with, five hundred maths problems that a Python interpreter helps with. Then markers get injected into one half only — text that is causally irrelevant to whether a tool is needed, the kind of decoration a real prompt template carries: a provenance tag, a citation marker, an output-format instruction.
After training, the agent is tested on maths questions, which never needed search. Each question is asked twice: once with the marker, once without. The gap between those two is the whole measurement — same question, same difficulty, one string of decoration.
Every one of the three search-flavoured markers moves the tool decision, and the size depends on which maths set is asked. The important property is that a difference exists at all: the policy is reading the wrapper, not the problem.
The same experiment run the other way round mostly does nothing. Code-flavoured markers — a notebook cell header, a shebang line, a generic code tag — injected into the maths half barely move Python calls on factual questions. That asymmetry is the paper's actual finding, and it is not about the markers.
During training, accuracy on the factual half climbs from roughly 45% to roughly 75%. Accuracy on the maths half stays flat around 45-50% — the model never really learns to solve those with the interpreter. Shortcuts form on the side that is working. Switch the arms.
A third arm settles what is doing the work. Pair a code-flavoured marker with the factual task — the one the model is competent at — and the shortcut comes back weakly, at most 3.5 points. Competence appears to be what makes a shortcut possible; how well the marker matches the tool's semantics decides how strong it gets. The practical reading: your best-performing tool is the one most likely to be over-called.
The obvious follow-up is how reliable the pattern has to be before the policy adopts it. The authors varied the share of training examples in which the marker and the tool appear together, from perfect correlation down to a coin-flippish 60%, and retrained at each level.
The answer is not a smooth slope. Drag the correlation.
Between 80% and 70% the effect collapses from double digits to zero. That threshold is a property of this setup, not a law — but it tells you what kind of data hygiene matters. A template that accompanies a tool nine times in ten is a training signal. The same template at seven in ten is noise the policy ignores.
The mitigation is one line of reward shaping. Alongside the usual reward for getting the task right, every individual tool call is shown to a small judge model with the original question, the tool, and the arguments. The judge answers one binary question — could a competent model have answered this correctly without the tool? — and each unnecessary call costs a flat 0.5.
The judge is not a filter at inference time. It shapes the policy during training, so the finished agent calls fewer tools without anything sitting in the request path. Toggle it on.
Six marker-and-dataset combinations, every one of them driven to within 0.3 points of no effect. The judge here is a small, cheap model, and the authors are careful to call it a practical design choice rather than a standard — its own biases become the policy's biases. But as a training-time regulariser on tool use, it costs a fraction of a step and it removes the entire effect.
Most teams do not run reinforcement learning on tool policies. The mechanism still reaches you, because the same correlation lives in the places you do control: the few-shot examples that always show the retrieval tool alongside the same header, the router trained on logs where one customer's formatting dominates one tool, the system prompt that pairs a tool with a phrase.
The cheap audit is the paper's own counterfactual: take real prompts, strip the decoration, and see whether the tool decision moves. Pick a pattern and set how consistently it co-occurs with the tool in your data.
If the gap is real, you have two levers and both are cheap. Break the correlation — vary the formatting across your examples until no template co-occurs with a tool more than a few times in five — or add the necessity check, as a judged label in an offline eval rather than a training reward.