The instruction file beat the code

Three papers about the configuration layer you wrote and then stopped looking at: the instruction file your coding agent reads before it reads the code, the minimum count sitting in your extraction schema, and the memory system you built instead of a retriever.

Today's pick

4.2% → 91.4%

convention choices that followed the existing code rather than the stale instruction file, before and after adding one sentence

Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix

Your project instruction file drifts — it still says use print() for progress messages while the code moved to a logger a year ago. Across 40 Python projects built so the file and the code disagree on four or five choices, and 320 unattended feature runs by two mainstream coding agents, 90.1% of those choices followed the stale file and 4.2% followed the code. Noticing does not help: one agent named the conflict in 79 of 80 runs and still went with the file in 86.2% of its choices, while the other silently rewrote working code in 20 of 80 runs, deleting a median of 198 lines and breaking the project's original tests in 17 of them. The authors read this as a precedence problem — the file reaches the model as an instruction and the code only as tool output, which models are trained to trust less — so one sentence telling the agent to follow the code and report the inconsistency moved code-following to 91.4% and the test pass rate from 89.4% to 100%.

open the explainer → abstract PDF code

34.5% → 55.4%

extraction lists longer than the document's true item count, once the schema shows a minimum the decoder never enforces

Padding to k: A Minimum Count in the Schema Makes Open-Weight Models Pad Their Extraction Lists

If your extraction schema asks for at least three items and the document contains one, the model does not tell you it came up short — it finds two more. Across eight open-weight models, 558 real documents and nine annotated sets, merely showing a minimum count in the schema text raised the share of lists longer than the true count from 34.5% to 55.4%, and enforcing that minimum with constrained decoding took it to 90.1%. The padding is hard to catch because it is mostly real text lifted from the document: a guard that checks every entry appears in the source clears 46.6% of the extra entries. Two instincts make it worse — re-asking with the validation error pushed one model from 10.0% to 69.0% over-count, and writing "exactly k" in prose reached 50.9% — while an explicit line saying fewer is fine removes the average effect and costs recall, with true items missed rising from 17.0% to 24.3%.

open the explainer → abstract PDF code

6 of 7

benchmark-and-model settings where plain retrieval over raw chat history beat every purpose-built memory system

Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought

Before building an agent that extracts atomic facts, summarises sessions and maintains a graph over conversation history, check what the dumb version scores. Chunking the raw history into 256-token windows, retrieving the 128 nearest by embedding and reranking down to 15 ranked first against twelve purpose-built memory systems in six of seven benchmark-and-model settings, by margins of 0.13 to 5.53 points, and lost the seventh 43.19 to 44.71. The reason is where the variance actually sits: across systems, how often the retrieved evidence was sufficient at all ranges from 19.07% to 45.29% on one benchmark, while accuracy given sufficient evidence barely moves, 80.00% to 89.71%. Extraction pipelines discard the detail nobody anticipated needing, and they are not cheap — 0.33 hours to ingest a million-token history against 26.95 for the slowest system, and 21.6 to 109.3 million tokens of extra model calls where the baseline spends about twelve minutes.

open the explainer → abstract PDF