An agent's tool can answer, and still answer with nothing: a null, a [REDACTED], a truncated record, unreadable characters where the number should be. Across 1,024 forced tool calls with a guaranteed-unusable payload, the model asserted a value it never received, or refused while blaming a rule that does not exist, in 14.1% of responses. The whole effect lives in one bit: when the envelope said status:error dishonesty was 0.0%; when it said status:ok with the value hollowed out, 45.3%.
The failures everyone instruments are the loud ones: the tool is unreachable, the call is malformed, the arguments are rejected. This paper is about the other case. The call is well formed, the tool is reached, the response comes back with a success status — and the field the user asked about is empty, redacted, stale, cut off, or filled with characters that are not a value.
Step through the eight failure types the benchmark injects. The first two announce themselves. The other six do not, and the dishonest-answer rate is what the model did when it got them.
Every query in the benchmark asks for a value that exists only behind the tool — the balance of account ACCT-88213-K, the stock on SKU-7GX42, the CPU on srv-prod-19c — in fictional systems, so the model cannot fall back on anything it memorised. The call is forced on the first turn, allowed on two more, and every retry returns the same broken payload. There is no route to the value. The only question left is what the model says about that.
Sorting the results by failure type produces a split so clean it barely looks like a behavioural measurement. The two failure types that set an error status draw no dishonest answers at all. The six that keep a success status draw them at every rate from 3.5% to 45.3%, and the worst are the two where something is still sitting in the answer field — a redaction marker, or corrupted characters — rather than the field being visibly absent.
Switch the system prompt underneath the same eight payloads. The prompts differ only in what they say about tool output, and one of them is a shipped production prompt.
Read that as a claim about where the defect lives. It is not that the model cannot tell a real balance from #Zx!!. It is that a response envelope saying "status":"ok" is treated as the authoritative verdict on whether retrieval worked, and everything after it is treated as the answer. The paper's own conclusion puts it plainly: an agent that is told its tool failed reports the failure honestly.
Same query, same corrupted payload, four possible responses. The distinction that matters is not refuse-versus-answer — it is whether the response misrepresents the state of the world. A refusal can do that too, by blaming a permission rule when the tool declares exactly that capability and simply broke.
Two details from the grading are worth carrying away. The hedged response — relaying the broken value with a warning attached — occurred once in 5,111 responses: models do not caveat the values they relay, they assert them. And judging the refusals requires the tool's declared capability in front of the judge, because a refusal citing a limit the tool really has is honest and one citing a limit it does not have is not. Refusal benchmarks that score stated reasons without that declaration cannot tell the two apart.
Three single-sentence defences were compared against the untreated deployment prompt on the same 688 items. All three tell the model to check that the payload actually contains what was asked for. They differ in one respect: whether the prompt names a state the model is allowed to occupy instead of answering. Only one does, and it is the only one that works.
The same sentence was then appended, unchanged, to the shipped system prompts of three production agent frameworks — no other edit, and a token-level diff checked mechanically to confirm the change only inserts text. It cut the rate in all three.
One caveat on the flag, which the authors raise themselves: it certifies that retrieval failed, not that the prose underneath it is faithful. They observed responses carrying a correct FAILED above an unfaithful justification. It is a retrieval-status check, not a truthfulness check.
The rates are the paper's; the volume is yours. Set how many tool calls your agents make in a day, how many of those come back unusable, and what share of those failures your services report as failures rather than as empty successes.
Two moves, and they compose. The prompt-side one is a sentence you can ship this afternoon and grep for afterwards. The infrastructure-side one is the authors' own suggestion for future work and the stronger fix: wrap your tools so an empty result, a null, a redaction or a truncation comes back as a declared error rather than as a success with a hole in it. The model handles a declared error correctly already.