The retriever kept choosing the tool that was down

Three papers about checks that pass while the thing you shipped is wrong: the retriever that ranks a dead tool first, the generated call aimed at the wrong server, and the correct patch that does twice the work.

Today's pick

86.2% → 2.9%

share of an agent's tool calls that went to a failing tool during a 30% outage, before and after execution feedback

RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics

Your tool index ranks on how well a description matches the request, which says nothing about whether the tool answers when it is called — and in a crawl of 5,172 tools from four agent frameworks, only 56.8% passed an automated check for working links and credentials. When 30% of the pool was made to fail mid-run, an embedding retriever kept sending 86.2% of calls to a tool that was down, because a dead tool is still the best semantic match; keeping one success-or-error bit per tool and letting it re-rank the top ten cut that to 2.9%, at 0.17 ms added per call. The learner is deliberately small — a per-tool linear model over the query embedding, discounted so old successes decay, weighted 15% against 75% for similarity — so nothing retrains when an endpoint starts erroring. The ranking gain is narrower than the headline suggests: 52.2% better mean reciprocal rank than the tool retriever it replaces, but still behind an off-the-shelf embedding model on real queries, so the part to copy is the feedback loop, not the encoder.

open the explainer → abstract PDF code

3.76%

generated service calls that hit the right host, path and method working from the request alone

SCCGBench: Executability-Oriented Evaluation for Real-World Web Service-Call Code Generation

Ask a model for the code that calls a web API and it hands you something that imports cleanly, builds a well-formed request 81% to 92% of the time, and points at the wrong server. A benchmark of 3,135 real API-calling functions, mined from 2,082 repositories and re-described so the host and path were stripped out of the prompt, ran seven models against a fake network that records the request instead of sending it: working from the request alone, the best model got host, path and method all correct 3.76% of the time, and wrong hosts account for 69.8% of every failure — the model substitutes an API it has seen for the one you meant. Hand over the service's own documentation and the same models reach 88.7% to 90.5%; retrieve it rather than hand it over and you get 27%, because the retriever surfaces the right page for about a quarter of queries. Pin the spec for the service you actually call into the prompt, and treat a search across your doc corpus as the weakest of the three options.

open the explainer → abstract PDF

42%

resolved issues where two correct patches differed measurably in how much work they do

Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution

A coding agent's patch passes the issue's test, so it merges, and nobody measures what it costs to run. Reusing each issue's own failing test as a micro-benchmark — warm up, then count hardware instructions over five replicates — a team compared the distinct correct patches for 262 real issues and found a statistically significant performance gap in 42.0% of them, with wall-clock time agreeing on 34.7%. The rankings scramble: the model that resolved the most issues (87.3%) sits mid-table on runtime, and the second-best resolver wrote the only patches consistently slower than the maintainers' own, at 101.3% of their instruction count. In one case three strategies all passed the same test while one quietly reinstated a heavy import the maintainers had deliberately dropped: 103.4 billion instructions against 48.8 billion. If agents merge on green in your repository, put an instruction-count or timing check on the path the patch touches, because the test that proves correctness says nothing about this.

open the explainer → abstract PDF