A coding agent that remembers the last ticket should do better on the next one, and on 111 real repository tasks the experience was genuinely worth having: handed to the agent directly it lifted resolution by up to 4.5 points and cut steps on every model tried. Then four off-the-shelf memory systems were asked to build and retrieve that same experience from the same history, and 11 of 12 pairings finished below the agent with memory switched off. Almost none of that was a retrieval miss. 69.3% of the failures found the right history and stored it in a form that no longer contained the fix.
Most memory benchmarks score whether a system can recall a fact. This one asks a harder question: does what it remembers make the agent fix the bug. The authors mined 3,634 completed runs from 90 repositories, and kept a task only if injecting a specific piece of that history demonstrably improved the outcome under a reference model, decided by the repository's own tests. So every one of the 111 tasks ships with a piece of experience already proven to help. Nothing here depends on the experience being valuable; that was established before the experiment started.
Hand that proven experience straight to five models that never saw the filtering, and see what it is worth.
The gains are small and one model shows none at all, which the authors are careful about: the confidence intervals cross zero. The robust part is the other column. Every model took fewer steps with the experience in hand, and four of the five spent fewer input tokens doing it. Whatever else is true, the experience is real information and the agent uses it.
Now the actual test. Same targets, same history, same solvers — but instead of handing over the verified experience, four existing memory systems are left to construct it from the raw trajectories and retrieve it themselves. This is what you are buying when you add a memory layer to an agent: not the experience, but a system's guess at which experience and in what form.
Each bar is one solver paired with one memory system, measured against that solver with memory switched off.
One pairing out of twelve finished above the baseline, by 2.0 points, and its interval crosses zero too. The clearest signal in the whole table is in the other direction: one pairing sits entirely below zero.
The obvious reading is that memory systems are bad at finding things. The paper closes that door in the next section, and the real answer is more useful.
The authors took 231 failing pairings over 85 targets and walked each one through the four places a memory pipeline can break: the history never got stored, it got stored but never retrieved, it got retrieved in a form that had lost the useful part, or it arrived intact and the run failed anyway. The distribution is lopsided enough to change what you would build.
Walk the pipeline stage by stage.
Retrieval is almost never the problem. Storage coverage accounts for 16.0% and ranking for 1.3%; together they are under a fifth of the failures. The overwhelming majority — 69.3% — are cases where the system reached the right prior run and wrote it down in a shape that no longer carried the change that mattered. Every retrieval benchmark in the field scores the two stages that are working.
Inside that 69.3% the failures sort cleanly by how the system decided to store things, and the two popular designs fail in exactly opposite directions. One keeps the raw conversation and carries the noise along with it. The other distils each run into tidy standalone facts and distils the specifics out.
Switch between the two designs.
The most interesting result in the paper is a small ablation. On the 62 pairings poisoned by stale instructions, deleting the flagged lines recovered 0.60 seeds per pairing — but deleting the same number of lines at random recovered 0.48. The harm is mostly the sheer volume of transcript, not the specific instructions inside it. A memory layer that pastes back the last run's conversation is spending your context window to make the task harder.
The experiment you can copy is the one the authors built the benchmark around, and it takes an afternoon. Find one task where you know a previous run contains the answer. Paste that prior run's useful part straight into the prompt and confirm it helps. Then let your memory layer do it, and compare what comes out against what you pasted in.
The benchmark's own task mix is a reminder of what this is about: 29.7% bug fixes, 20.7% feature additions, 14.4% parser and serialisation work. These are jobs where the useful prior knowledge is a specific diff in a specific file, which is precisely the thing both storage designs are built to discard.