Compaction keeps the goal, drops the rules

Three papers on one failure: the fact that decides the answer exists only in the context, and the layer in front of it quietly drops it — the summary that keeps the goal and loses the rules, the tool output that prints what it found but not what it capped, and the model's own account of which source it used, which never consults the document at all.

Today's pick

35.9% vs 6.2%

of a user's how-to-work instructions dropped by the next compaction summary, against descriptions of the feature to build, across 44 real sessions

Benchmarks Are Surprisingly Compaction-Friendly

Every coding-agent scaffold compacts: when the context fills, the run is summarised and carries on from that summary. On three long-horizon benchmarks that costs almost nothing, and the paper says why — the objective is fixed and the half-built work sits on disk, so most of what a summary drops can be read back. What is lost is the part with no copy anywhere else: the first summary was already missing 0.92 of a rebuild task's three hard rules, rising to 2.65 by the thirtieth, and across 44 real sessions 19% of the user's constraints were gone from the next summary — 6.2% of descriptions of what to build, against 35.9% of instructions on how to work and 36.8% on what not to touch. Keep those in a durable list and re-inject them verbatim after each compaction; none of five strategies beat plain summarising, and a small cheap summariser cost one model 20 points.

open the explainer → abstract PDF

90 of 90

verdicts in which a verifier accepted the same false answer, flipped to 0 of 90 by one added sentence in the tool's printed output

Who Verifies the Tools? How Warped Formatters Mislead Agents and Their Verifiers

A tool can compute the right result and print a sentence that misdescribes what it did: a date-range search fetched its 20-row maximum and the formatter counted the rows it received, printing “found 20 documents” when 4,772 matched. The agent told the user the archive holds exactly 20, and the verifier checking that answer against the tool output accepted it in 90 of 90 verdicts — it reads the same sentence and cannot recover the omitted number — while one added line stating the cap and the true total got the same false answer rejected 90 of 90 times. Across 40 cases, stating the search limit, the result cap, or that a value was estimated cut answers carrying a false claim by 62 to 92 percentage points; the fourth family, identifiers from two namespaces sharing one integer space, moved 6 points, so that one needs types rather than prose. Audit every place a tool result becomes text an agent reads, and judge correctness against retrieved facts rather than support from the output string.

open the explainer → abstract PDF code

0 of 12

model-and-edit combinations in which deleting or replacing the document changed the model's claim about which source its answer came from

Context or Knowledge: How LLMs Attribute Their Own Answers

Plenty of retrieval products log the model's own answer to “did you use the provided document or your own knowledge” and read “the document” as evidence of grounding. Delete that document from the cache once the answer is written and the report never moves — zero flips across four models and three edits — though the model can still quote a replacement document word for word in 100% of cases, even one stating precisely the answer it just gave. Copy the cached question-and-answer state from a run that saw a different document, changing no visible text, and the report follows the donor in all sixteen tests; one direction fitted to those cached values flips 98.9 to 100% of reports while the model goes on quoting the contradicting document. Applied as the answer is written, that direction switched the real source in 71% and 51% of cases against an instruction saying otherwise, so treat a self-reported source field as a fact about processing history and test claims against the retrieved span instead.

open the explainer → abstract PDF