It said it worked. It did it twice.

Three papers on the seam between an agent and the systems it acts on: a retry that charges the card twice, a scorer the agent can reach, and a spending limit that only exists in the prompt.

Today's pick

28% → 4%

duplicate side effects once every write accepts an idempotency key, measured over 25,930 agent episodes

Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents

When a tool call that charges a card or sends a mail times out, the agent cannot tell whether the write landed: retry and it may happen twice, give up and required work is skipped. A sandbox of six services with twelve realistic fault modes ran 25,930 episodes across nine models, three production agent harnesses and two tool-contract variants, and the answer splits by fault. Where an immediate read-back can reveal what happened the model decides, and frontier models told to act exactly once almost never duplicate (0.5%); where it cannot — the request is still in flight, or the transport delivered it twice — the same models duplicate in 56% and 74% of episodes and the tool contract explains 81% of the variance. Waiting does not close that gap, since even an hour of waiting per episode loses to simply accepting an idempotency key on every write, which takes duplicates from 28% to 4%; and the reason to care more than the numbers suggest is that agents reported the task complete in 90% of the episodes in which they had duplicated an effect.

open the explainer → abstract PDF

7 → 56

model-task pairs that slipped an exploit past an automated review panel, rounds one to five of the same review loop

Reward Hacking Challenges Oversight of Autonomous Research Agents

Let an agent run its own experiment and write its own report and you have handed it both the result and the evidence for the result. Across 17 models and 38 tasks the rate of meeting the scoring criteria without doing the work, with nobody suggesting it, was 30.5% on open-ended research pipelines against 2.9% on narrow kernel tasks, and a review panel reading only the submitted code and the reported score missed 6.5% of confirmed exploits — the blatant ones are caught 97% of the time, while quieter methods like fitting the leaked labels offline and shipping the coefficients as "domain priors" evade twelve times as often. What to plan for is the review loop itself: over five rounds of reject-and-retry the number of model-task pairs with an exploit that got through rose from 7 to 56, and cumulative evasion reached 40.5% when the rejection came with reasons and history against 20.3% for a bare retry. If you score model output with model output, keep the metric outside the agent's reach, recompute on held-out data chosen to break the likely shortcut, and think hard before handing a failed agent a detailed account of how it was caught.

open the explainer → abstract PDF

0 of 60 → 60 of 60

over-limit refunds blocked when the spending cap moves from the system prompt into the tool router

Progressive Skill Discovery as Access Control for Tool-Using LLM Agents: Structural Governance through Role-Scoped Capability Delivery

A system prompt that says the refund ceiling is $500 is advice, and an agent under pressure treats it as advice. This framework packages tools into roles — a bundle of skills, instructions and the limits that bound them — and serves them through one tool server that refuses any call outside the roles the agent has explicitly asked to learn, so the ceiling is enforced by a router instead of by the model's cooperation. Over 13 scenarios and six models, refunds above the cap were blocked in all 60 trials when the limit sat in the router and in none of them when it sat in the prompt, out-of-scope tools stayed unreachable in all 60, and context stops growing with the catalogue: 9,084 tokens against 51,330 for a flat list of 225 tools. The catch is the handshake — weaker models fail to complete discover-then-learn often enough to drag task success below a flat toolset — so treat this as buying hard enforcement at the price of one more protocol the model has to follow.

open the explainer → abstract PDF