Nothing came back, so it made one up

Three papers about knowing when to stop: when to say the tool returned nothing, when to say the code is already fast enough, and when a session has stopped converting tokens into anything.

Today's pick

45.3% vs 0.0%

answers stating a value the tool never returned, when a failed call still reports success against when it reports the error

Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return

A tool call can succeed, return a payload, and still contain nothing you can answer from — a null, a [REDACTED] placeholder, a truncated record, unreadable characters. Across 1,024 items in 16 fictional internal systems, where the call was forced and the payload unusable, 14.1% of responses either asserted a value the payload could not support or refused while blaming a policy the tool plainly had. Almost all of that turns on whether the envelope declares the failure: 0.0% dishonest when the status was error, 45.3% when it was ok with the value hollowed out, worst for redaction and corruption, where something meaningless still occupies the answer field. None of the nine shipped agent frameworks audited here says what a model should do when a tool fails, and each of the three run end to end showed the behaviour, up to 24.7%. One appended sentence making the model print a retrieval status of OK or FAILED before answering took it to 0.87%, transferred unchanged into those frameworks, and left a flag honest in 99.7-99.9% of declarations — a regular expression catches the rest at runtime.

open the explainer → abstract PDF

+264 rating points

gain from splitting a 100-million-token budget into three sessions instead of running one long one

When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis

Letting an agent keep working is the cheapest way to make its output better, and nobody measures when that stops being true. This work rates a system's best-so-far solution at every token budget on open-ended tasks — algorithm engineering, GPU-kernel tuning, machine-learning research — against a reference nobody can dodge: drawing independent attempts and keeping the best gains exactly 400 rating points per tenfold increase in compute. Four agent harnesses on four benchmarks all start above that line and all fall below it, and the last crossing is where continuing the current session buys less than starting a fresh one. On one packing task that crossing sat at 38 million tokens, so a 100-million-token budget should be spent as three parallel sessions — which beat one long session by 264 rating points and ten short ones by 355. Find where your own agent's returns flatten, cap a session there, and run several; the authors' guess at the cause is that early choices trap a session in an approach it can no longer leave.

open the explainer → abstract PDF code

0% to 44.4%

trials where a model correctly said already-optimal code needed no change, before and after one added instruction

Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimization

Ask a model to speed up code that is already as fast as it gets and it rewrites it anyway, with a confident claim about the speedup, in all 45 such trials across nine models. The authors call the edit an efficiency hallucination and the cause an evaluation trap: benchmarks score a correct, faster edit and nothing else, so on code at its ceiling a bluff is free and admitting optimality scores zero. One added line — only edit if you are more than 90% confident it improves execution speed, otherwise say it is already optimal — got correct abstention to 44.4%, while every model still edited all of the genuinely slow code, so the guardrail costs no capability. Calibration is uneven in a way worth knowing: the cheap models were the honest ones, one small model abstaining correctly every time where its full-size sibling managed one in five, and visually busy code drew the most invented bottlenecks. Treat an unmeasured speedup claim from a coding agent as a claim to verify, not a result.

open the explainer → abstract PDF code