The notes were written for a different reader

Three papers about components that were measured alone and then deployed in company. A memory store calibrated to the model that wrote it, a reviewer that is the same model as the executor, a confidence guarantee calibrated on a model answering by itself — each holds up in isolation and quietly stops holding once something else is in the room.

Today's pick

13 pts vs 0.04 pts

accuracy the agent loses when a new model reads the old one's memory — model-written notes versus a fixed-schema store

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

Upgrading the model behind an agent is routine; migrating its memory is not, so the store stays and the new model is simply pointed at it. Holding 48 identical histories fixed and varying only how they are stored — full transcript, retrieved chunks, model-written notes, or subject–predicate–object facts under a fixed schema — this study swaps the model that reads them. Notes are the fragile case: accuracy moves −13.28 points in one migration direction and +9.91 in the other, so the same upgrade is a loss or a gain depending on who wrote and who reads, while the fixed-schema store shifts by 0.0004 and re-embedding half a vector index captures only 4.96 of the 11.90 points a full re-embed gives. Notes also cannot be repaired from the notes — store-only repair missed the target in all 48 cases while a retained raw history recovered 34 — so keep source transcripts and test each upgrade direction on its own.

open the explainer → abstract PDF

35% vs 2%

correct answers the reviewer threw out — the executor checking its own work versus a different model family

Reviewer Capability Governs Rejection Targeting, Not Repair Skill

Execute–review–revise pipelines put a cheaper model in the reviewer seat to avoid paying flagship rates at every stage, and this pilot varies who sits there while holding the executor fixed across 100 competition maths problems. A mid-tier reviewer from a different family lifted final accuracy from 52% to 64% with zero answers damaged; the executor reviewing its own work caught more errors than any other condition (0.85 recall) and gained nothing significant, because it rejected 2.1 times as often for a third the repair rate and falsely rejected 35% of its own correct answers against 2%. Its apparent safety turns out to be an accident of the executor ignoring criticism: of 18 correct answers it wrongly rejected, the executor left 15 alone and revised 3, and all 3 went wrong — a pipeline that actually complied would land below no review at all. Below a capability floor the seat is inert; the weakest reviewer changed none of 100 final answers while doubling token spend.

open the explainer → abstract PDF code

90% → 74%

coverage a calibrated confidence set actually delivered once six peers asserted the same wrong answer

Conformity Breaks Conformal Prediction

Conformal prediction turns a model's scores into a set of candidate answers with a promise attached — the true answer is inside 90% of the time — which makes it an attractive gate for deciding when an agent may act alone and when it must escalate. Calibrate that gate on a model answering by itself, then place it in a discussion where six peers assert the same wrong answer, and delivered coverage falls to 73.9%; on the decile just above the threshold, which an attacker can pick out without knowing the answers, it falls from 87.3% to 47.4% while the overall number a monitor watches still reads 75.7%. The guarantee breaks in a way the standard repairs cannot reach: the question distribution never changed, only the model's scoring did, so importance-weighting reduces to the identity and per-model calibration still leaves three of four models under target. If other agents' text reaches your model before it scores an answer, calibrate under that condition, because a clean certificate describes a situation your system is never in.

open the explainer → abstract PDF code