Splitting the agent cost 28 points

Three upgrades, each finally measured against the baseline it needed: a multi-agent split against one agent on the same token budget, a cheap decision model against a placebo, and agentic retrieval against a single retrieval round. None paid.

Today's pick

-28 pts

average loss for a multi-agent split against one agent on the same model and the same per-conversation token budget

Does Multi-Agent Collaboration Pay When Agents Serve Users? Testing Agent Scaling in Dual-Control Settings

Split a policy-breaking agent into a planner, a tool-caller and a policy checker: a widely-cited rule says that pays while the single agent scores under about 45%. That rule was measured with nobody on the other end, so this study re-ran it in conversation — a customer who volunteers information turn by turn under a written airline or telecom policy, both systems built from the same model and charged to one shared per-conversation budget of generated tokens. Across 7,950 conversations the split version lost in 12 of 13 model-and-domain cells and tied in the last, 28 points worse on average; remove the budget ceiling and it only draws level at 5.5 times the token spend. Three quarters of the loss lands in conversations that ran out of budget mid-task, wrongful database writes never measurably fell, and one agent running the same plan-then-review cycle in a single context held its ground — the cost was dividing the work, not the discipline.

open the explainer → abstract PDF code

+19%

cost per attempt at an identical 73.0% pass rate, with a placebo matching that pass rate for free

Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

Scaffolds increasingly hand the small calls — stop now, verify first, change approach — to a cheap model, and the evidence offered is that its judgements are accurate. This study held one terminal-agent loop fixed, varied only that controller, and added the baseline the claim needs: a placebo interrupting at exactly the same moments with choices drawn from the real controller's own answer frequencies. Its judgement was genuinely better than hand-written rules or a cheap chat model, and it still raised cost per attempt by 19% at an identical 73.0% pass rate — 22% dearer than the placebo, which matched that pass rate for nothing. The fault is calibration, not ranking: the agent's own claims of being done were right 87% of the time while the controller's average confidence in them sat at 0.50, so the money went on re-checking correct work. Set such a gate's threshold from the base rate you measure, and give it a placebo arm before believing it earns its keep.

open the explainer → abstract PDF code

56% to 46%

correct refusals once the model may judge its own context and fetch more, for a 0.8-point accuracy change

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

A retrieval system is trustworthy only if it can say the answer is in neither the documents nor the model, and accuracy hides that. Asked directly — 300 questions whose answers were documented only after the search corpus was frozen, so no honest answer exists — one round of retrieval is a large honesty win: correct refusals rise from 17% without retrieval to 56% with it, accuracy on the answerable questions from 24% to 69%. Letting the model then grade its own context and fetch up to two more rounds, the agentic loop most teams reach for next, drags correct refusals back to 46% — lower in 30 of 40 model settings and higher in none — while accuracy moves 68.5% to 69.3%. The mechanism is substitution: given passages that look relevant but describe a neighbouring thing, the model answers about that. Hold out questions your corpus cannot answer and score abstention separately, because every extra retrieval hop is a hallucination regression accuracy hides.

open the explainer → abstract PDF code