Anything that caches facts about a codebase has to decide, on every commit, which of those facts just went stale. The usual signal asks a model whether the diff preserves behaviour — a question about the change, not about anything you stored. Five models spanning a 40× price range fired on 59–72% of real commits at precisions of 0.291 to 0.329, against a base rate of 0.25. Asked instead whether one specific stored claim still holds, on the same diffs, the same models reached 0.705 to 0.974. Handing the first judge the claim text without changing the question moved precision by 0.010; changing the question moved it by 0.49.
Give an agent a memory of your repository and you have built a cache, which means you have inherited the cache’s hardest question: what just went stale? The cheap answer is to invalidate anything whose source file was touched, and it fires constantly, because most commits touch files that most stored facts came from. The expensive answer is to ask a model whether the commit preserves behaviour, and it sounds like the right question — a change that preserves behaviour cannot have broken anything.
It is not the right question. Preserving behaviour is a property of the diff, and almost no real commit preserves behaviour in the strict sense — that is what a commit is for. Asked that question on 604 held-out claims where one in four had genuinely broken, five models fired on between 59% and 72% of them. The same models, on the same diffs, asked whether one particular stored claim still holds, fired on 17% to 29%.
Switch the question below and watch the alert volume, not the accuracy.
The fire rate is the whole tell. Both questions are answered by the same model at the same temperature with the same diff in the context window. Only the thing being asked about changed, and the alert volume fell by two thirds.
If the behaviour-preservation question were merely hard, a better model would do better at it. The paper runs the ladder: five models from a tier costing $0.025 per million tokens up to a frontier model roughly forty times the price. On the diff question every one of them lands between 0.291 and 0.329 precision — the ladder is flat, and flat at barely above the 0.25 you would get by invalidating everything. On the claim question the same ladder climbs from 0.705 to 0.974. Capability only starts paying once the question is answerable.
Four of the five models are within 0.04 of each other on the diff question. On the claim question the same five span 0.27. A question that every model answers equally badly is not a hard question; it is the wrong one.
The control is what turns this from an observation into a finding. If the claim-relative question wins because the model gets to see the claim text, then handing the claim text to the behaviour-preservation judge should close the gap.
It does not close the gap: precision moves by 0.010 and 0.016. Changing the question, with no new information at all, moves it by 0.49 and 0.65. The information was never the bottleneck.
The reason is visible as soon as you work an example. A refactor that renames an internal helper changes behaviour for anyone reaching into it, so the diff question says not preserved and every stored fact is thrown away — while a claim about the public return value is untouched. Run it the other way and a one-character change to a default argument looks like housekeeping while the claim is dead.
The reason invalidation benchmarks tend to be annotation exercises is that judging whether a fact went stale is exactly the thing under test. This paper refuses the loop. A claim is a test function together with its assertions, passing deterministically at one commit. It has flipped if that same assertion text fails at the next. Nobody labels anything; the label is whether the assertion ran green or red.
That definition runs straight into a wall, and the wall is the interesting part. Step through how the corpus is built.
On a CI-gated mainline, a commit that leaves a pre-existing test failing cannot merge — so if you look for unmodified tests that broke, the positive class is empty by construction. The fix is to evaluate the parent’s assertion text against the child’s source: restore the old test file at the new commit and run it. The assertions that now fail are the claims the commit invalidated, whether or not the author updated the test in the same breath.
Two base rates come out of this and they are easy to confuse. The natural rate a deployment faces is 1.8% — 184 flips in 10,369 claims. The precision figures above are measured on a stratified split where the rate is 0.25, which is why 0.29 precision is barely better than invalidating everything rather than merely mediocre.
The strongest non-model baseline here is not a heuristic, it is a deployed tool: a regression-test selector carrying coverage-derived dependency data, so it knows with near-certainty which tests a given change can possibly touch. That is the best case for the reachability idea, and it reaches 0.868 recall at 0.415 precision. Complete knowledge of what a change can reach turns out not to identify what it falsifies, because reachable and falsified are different sets and only one of them is what you wanted.
Step through the signals and watch where each one sits.
Plotted against each other, this is not a frontier you trade along. The claim-relative question sits up and to the right of the anchoring signals at comparable recall to the diff question, which means there is no budget argument for the worse signal — it is not buying recall with its false alarms.
Put numbers on it. You keep some thousands of cached facts about a repository — retrieved summaries, an agent’s notes, an index of what each module does — and the repository takes some number of commits a day. Every invalidation you raise costs a re-derivation; every flip you miss is a stale fact served to a model as if it were current.
The diff question does not merely waste money, though it does that. The worse failure is what the noise does to you afterwards: a signal that fires on two thirds of commits is one you will start ignoring, batching or sampling, and every one of those coping strategies converts false alarms into missed flips.
The claim-relative question is not free either — it needs one model call per candidate claim against the diff, rather than one per commit. That is the trade: the per-claim call is small and cacheable by claim, and what it buys is an alert volume a job queue can actually act on.