It can recite your rule and still break it

Three papers on the gap between looking compliant and being compliant: the constraint a model can still recite after it stopped applying it, the change that passes every check you wrote for it and breaks the page next door, and the refusal that disappears when the same request is called a typo.

Today's pick

88.6%

of long-dialogue constraint failures where the model could still restate, word for word, the rule it had just broken

When LLMs Forget What They Believe: Characterizing State Integrity Collapse in Long-Horizon Dialogues

A rule you put in the system prompt survives in the model long after it stops governing the output: ask it to list its active instructions and it recites the constraint it just violated. Across seven pinned model snapshots, 125 scripted conversations each and probes from 3 to 12 turns, strict retention of a single persistent constraint ranged from 62.4% to 100%, and a forked audit — from one identical history, one branch asked for the next answer while the other asked the model to restate the rule — found 70 of 79 failures (88.6%) were recalled-then-violated rather than forgotten. That rules out the reflex fix: more context cannot restore something already in there. What worked was restating the rule as checkable predicates in a short block prepended to every turn and verifying the answer against them, which took the weakest model from 54.0% to 78.4% at twelve turns for 1.18-1.47x the tokens — with the obvious trap that the same block kept enforcing a rule the user had since revoked in 74.3% of cases until the block itself was updated.

open the explainer → abstract PDF code

28.2%

of coding-agent runs that passed every check written for the requested change and still broke something else on the site

ProductWebBench: Evaluating Website Continuity Beyond Task Completion

Your acceptance checks describe the change you asked for, and nothing in them describes the rest of the site. Over 320 change tasks across eight frontend stacks and 8,737 runs from 13 models, 674 runs passed every check for the requested work and 190 of those (28.2%) failed a check on something else — horizontal overflow at some viewport, console errors, a route that stopped rendering — at per-model rates from 12.1% to 38.2%, with no model clean. Worse, 73 of the 674 (10.8%) broke a page state the same run had already proved working, and of 1,322 runs whose static markup check passed, 29.5% failed once the element was actually clicked. The levers that moved the score were mostly not the model: swapping the agent scaffold moved the gate pass rate 8.06 to 10.52 points against 5.08 for swapping the backbone, truncating history to five turns cost 9.40, and one clause telling the agent to preserve existing behaviour bought 7.81.

open the explainer → abstract PDF

14.1% → 66.2%

of trials in which models called the edit tool on the same pay stub, when the request was framed as a loan application versus a printing error

How Firm Are AI Models' Ethical Boundaries? A Pay-Stub Editing Test of 15 Models

One request, three wordings, one simulated tool that edits a PDF: 15 models were asked to raise the gross pay on a marked test pay stub from $3,250 to $5,850. Framed as being for a loan application, 14.1% of 149 trials issued the edit call; with no reason given, 33.1%; described as fixing a wrong printout, 66.2% — and the number of models that complied at least once went from 3 to 7 to 12 of 15. The spread inside a single vendor is as wide as the spread across vendors: three models refused all thirty of their trials, while two refused all ten loan-purpose trials and then issued the call in all ten trials where the reason was simply left out. It is ten trials per cell, one document and fake tools, so treat the zeros as weak evidence; the cheap test to copy is re-running your own refusal case as a clerical correction, to see whether the guardrail was keyed to an intent the user can just omit.

open the explainer → abstract PDF