A catalog of typed tools is the default way to give an agent capabilities: declare the functions, declare the arguments, let the model fill them in. This paper measures that default against four alternatives — including simply handing the agent a shell — across 654 enterprise tasks and two frontier models. The shell on its own wins on quality and on cost: 69.4% against 44.9% on 174 simulated office tasks, at 264k tokens a task against 945k. Putting the typed catalog back on top of the shell, or letting the agent write and keep its own tools, adds nothing measurable.
The going assumption is that an agent does better when the actions available to it are described precisely. So you write a schema for each one — name, arguments, types, a sentence of documentation — and the model picks from that menu. It is the shape almost every tool integration ships in, and it is a real engineering investment: sixty hand-maintained functions in this paper's office environment alone.
The alternative the authors test is to describe nothing and hand over a shell. Switch between the two.
Two things move at once, and they move the same way. The shell scores twenty-four and a half points higher and spends about a quarter of the tokens. That second number is the one worth sitting with: the typed catalog is supposed to be the efficient option, because the model is not paying to discover anything. In practice its schema sits in the context of every single turn, and each result comes back as one more message to re-read.
The comparison is not two-sided. Five interfaces are run over the same environments, the same tasks and the same two models, and they span the space between a closed catalog and arbitrary execution — including two designs that try to have both. Step through what each one actually hands the agent.
Two of those five are attempts at a compromise, and they are the interesting ones. Shell plus catalog gives the agent both and lets it choose. Programmatic tool calling keeps the catalog closed but lets the agent write a small program over it in a runtime with no shell and no other way out — the design you reach for when a compliance rule says the set of possible actions must be enumerable in advance. It is genuinely cheaper than calling the typed tools one at a time. It is not as good as the shell.
Two environments, deliberately unalike. The first is a simulated software company: a self-hosted code host, a file store, a project tracker, a chat server and seventeen simulated coworkers, scored by checkpoint scripts. The second is 480 professional-services tasks — investment banking, management consulting, corporate law — scored by a rubric over the final answer and the files left behind. Switch the metric and the model.
The gap is much bigger on the office tasks than on the professional-services ones — twenty-four points against six. The authors read that as flexible execution helping most where the work is moving between systems and least where the work is domain judgement, and that reading matters for whether this result is yours. If your agent's hard part is knowing what a good answer looks like, the interface will not save you. If its hard part is getting at four services and a filesystem, it might be most of the problem.
One caution the paper is explicit about: these are two model versions at one moment. The recommendation is not guaranteed to survive the next release in either family.
The token result is not a rounding artefact, and the mechanism is visible in the call logs. A typed tool call does exactly one declared thing and returns one result to read. A shell call is a line of script, so the agent bundles: pipe, loop, redirect, chain. Step through what the two call streams actually look like.
The repetition line is the sharpest of them. With the typed catalog, 16.2% of calls on the office tasks repeat a call the agent already made — the same function, the same arguments, the answer already in its context. Give the same model a shell and that falls to 0.3%. Repeated calls are a context-management failure, and the shell appears to make them nearly impossible to commit, probably because a command and its output read as one block rather than as a structured result the model has to remember it already has.
Which leads to the obvious next idea, and to this paper's most useful negative result. If bundling is what helps, let the agent bundle permanently: give it a persistent directory, tell it to write reusable scripts, and let each task start from the tools the last one left behind.
The agent does the thing you asked. It writes the tools, it invokes about nine in ten of them, it reuses roughly half across tasks, and those reuses succeed at about the same rate as writing the code fresh. And on the tasks that actually invoked a saved tool, the median score change against a plain shell is zero — on both benchmarks. The library is real; the benefit is not detectable. Writing the code inline each time was already cheap enough that amortising it buys nothing.
Pick an interface and a daily task volume. Score and per-task token counts are the paper's measurements on Opus-4.8; multiplying them out to your traffic is arithmetic, and your tasks are not this paper's tasks — so read the shape, not the totals.
The two axes do not trade against each other here, which is what makes the result unusual. Normally you buy accuracy with tokens. In this comparison the cheapest interface is also the most accurate, and the expensive one is the one you hand-built. The authors' rule is short enough to act on: a shell wherever you can isolate arbitrary execution, and the restricted program-over-catalog runtime where a policy requires the set of possible actions to be fixed in advance.