Small open-weight models are assumed to fail at agent work because they cannot reason well enough. Eleven days of driving a real database client’s agent mode with 39 locally served models produced 8,199 runs and 2,100 model-attributed losses — and 75.7% of those losses came from runs that had already invoked a tool. The largest single class is a run that used its tools, did not run out of time, and never got a deliverable back. The production ledger recorded refusal codes but never the arguments the model sent, so for ten days that was invisible. Capturing them exposed five server-side defects; fixing those moved six models by 6 to 21 cells out of 30 without touching a model, a prompt or a sampling setting.
Everyone who has pointed a 7B model at an agent loop has the same story: it flails, it gives up, it is not there yet. That is a claim about where the failures are, and it is testable. The authors drove the agent mode of an open-source database client with 39 locally served models across six task surfaces for eleven days, logged every event, and sorted the 2,100 losses they could attribute to a model into four classes by what the run actually did.
Look at the classes, then regroup them by the only question that matters for debugging: did the model get a tool call out at all?
The authors are careful about which parts of that survive scrutiny, and you should be too. Resampled by model rather than by run — the right unit, because one chatty model can otherwise carry a finding on its own — the headline holds in 99.7% of resamples. The ordering of the four classes underneath it does not: transport stays largest in only 74.5%.
Here is the part that should make you open your own telemetry. The production system logged a refusal code for every rejected tool call — 14,008 of them — and never logged the arguments the model had actually sent. For ten days the failures were real, counted, and completely uninterpretable: a run would show a string of refusals and then a timeout, which reads in any ordinary dashboard as a model that could not follow the schema.
Turn argument capture on and step through what it found.
The fixes are unglamorous: recognise a call by its name rather than by parsing a strictly-shaped blob, read the statement kind from the statement instead of from an enum field, auto-rename a single surplus or single absent field when the intent is unambiguous, apply that rename on flat two-field calls, and stop telling models to remove a field that a sibling tool requires. Then the same sweep runs again — same models, same prompts, same sampling.
Flip the server version and watch the scoreboard move.
Two of those models had scored zero. Read on a leaderboard, a zero is a verdict on a model. Here it was a verdict on a field name. This is the paper’s real contribution to anyone shipping tool integrations: the measurement you publish about a model is a measurement of your harness and the model together, and you cannot separate them without capturing what crossed the boundary.
The authors then report, against their own interest, a confound they believe contaminates published local-model comparisons including their own. Served with no context cap, one model that is 7.1 GB on disk was admitted at its full 262,144-token window and sat resident at 51 GB on a 64 GB machine. Every run in that cell hit the turn limit. Capped at 32,768 tokens the same model held 5.1 GB and passed every run in the cell.
Slide the context cap and watch the machine fall over.
In any ordinary log those two configurations are the same model failing. The run ends at the turn limit either way; nothing in the event stream says “this box is paging”. The authors mark every per-model rate in the paper as a lower bound with unknown per-model bias for exactly this reason, because large-context models were penalised harder than small ones.
The transferable finding is not about small models at all. It is that a tool-calling agent has a boundary in it, and most teams instrument only one side. A refusal code tells you the server said no. It does not tell you whether the model sent something reasonable that your schema rejected for a reason you would be embarrassed by.
Switch on the fields your ledger captures and see how much of a failure you can actually attribute.
The shares in that panel are illustrative. The ordering is not: the single highest-value field is the one the authors were missing, the arguments the model actually sent, because it is the only one that can distinguish a model that cannot follow a schema from a schema that cannot be followed.