An agent cannot tell a complete answer from a well-formed fragment, because both arrive the same way: a valid response, no error, no flag. Auditing fifteen data tools behind one agent framework turned up 91 confirmed cases where the call reported success and the content was wrong or missing — a result set silently capped at 1,000 rows, a field the website shows and the interface omits, a filter the wrapper never exposes. Seven of fifteen tools failed more than half their test cases. 51 of the failures sat in the service interface and 25 in the wrapper around it.
Everything an agent knows about a tool call is the response. If the status is fine and the body parses and the fields it expected are there, the call worked. There is no channel through which a tool can say “this is most of the answer”, and so there is no channel through which an agent can hear it.
These five are from the audit. Each one is a real request against a real scientific data tool, returning a real response that an agent would have no reason to question.
The interesting thing about the distribution is how little of it is the model’s fault. Cases where the agent had a usable answer and misused it are rare. The mass sits in the service interface and in the adapter code that wraps it — one of which you can test against, and the other of which you wrote.
Per tool, the hit rate on targeted probes is high enough to be uncomfortable. Read these as “when somebody went looking, this is how often they found something”, not as a base rate on your traffic — the tests were written to target suspected gaps, and the authors say plainly that the proportions are not representative.
The paper’s one conceptual move is to stop asking whether a call executed and start asking whether what came back still supports the conclusion someone is about to draw from it. That splits into seven things a response can quietly fail at, and the audit counts how often each one showed up across the 91 failures.
Every one of these is a test you can write today against a tool you already ship, and each maps onto failures the audit actually found. Switch on the ones your wrapper asserts and see what is left uncovered.
Tests catch the gaps you thought of. The other half of the recommendation is a channel: let a wrapper say what it could not do, in a field the agent reads, so a limitation becomes part of the evidence rather than a thing nobody mentioned. Here is the same failure with and without one.