Bash beat the tool catalog

Three papers on the distance between what an agent does and what you can see: the interface you hand it, the check you run before it acts, and the summary it writes when it is finished.

Today's pick

69.4% vs 44.9%

task score with a plain shell against a curated catalog of typed tools, on 174 simulated office tasks

Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents

The standard way to give an agent capabilities is a catalog of typed functions with declared arguments — the shape most tool integrations ship in today. This study puts five interfaces head to head on 174 simulated office tasks and 480 professional-services tasks, across two frontier models: the typed catalog, the catalog plus a shell, a shell on its own, a shell that can save reusable scripts between tasks, and a sandboxed Python runtime allowed to call only the catalog. The shell on its own won on both quality and cost — 69.4% against 44.9% on the office tasks, at 264,000 tokens a task against 945,000 — and letting the agent build its own persistent tools on top of the shell added nothing measurable, a median of zero points on either benchmark. Before hand-authoring another tool schema, check whether the agent could reach the same system with a command instead; the authors' rule is a shell wherever you can isolate arbitrary execution, and the restricted runtime only where a fixed catalog is a compliance requirement.

open the explainer → abstract PDF

99.1% vs 0%

files silently corrupted after one line shifts, editing by line number against editing by matched content

Look Before You Leap: Pre-Action Verification for LLM Agents

A wrong action from an agent does not always throw an error — often it lands somewhere plausible and wrong, and nothing in the loop notices. This work pins down each action's intended effect before any executor runs, so those silent failures can be counted directly rather than inferred from downstream symptoms, and the sharpest result is about edit format: apply an edit by line number and a single line inserted above the target corrupts the file 99.1% of the time with no signal at all, while search/replace and unified diff fail loudly and cleanly at 0%. Editing a whole function by name is quieter but still lands on the wrong function 12.7% of the time on files nobody touched. Switch your agent's editor to a content-anchored format if it is not already on one, and consider a pre-flight check on shell commands too — three deterministic tests over 9,930 commands (does it parse, does the binary exist, is the flag real) caught 95.8% of invalid ones in about 2.5 milliseconds, the first two contributing half the catches at zero false positives.

open the explainer → abstract PDF

1 action in 11

share of what an agent did that its own closing summary mentions, over 5,851 real coding sessions

Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agents

When a coding agent finishes, what the developer reviews is a summary the agent wrote about itself rather than a display anyone designed. Across 5,851 real sessions and 355,942 tool calls, that summary referred to about one action in eleven, and a model given only the report reconstructed about a fifth of the action log — the median session ran 59 actions and produced a report making four claims. The uncomfortable finding is what the summary does when things go off script: reports track the executed log when execution followed the plan, but swing back toward the plan the agent abandoned as execution diverges from it, a rank correlation of 0.55. Thin reports are not a distress signal either — coverage barely moves when the session later needed human correction, about 1.6 points — so build your review surface from the action log and treat the closing prose as an index, not evidence.

open the explainer → abstract PDF