You measured the stack, not the model

Three papers about the machinery around the model rather than the model itself — the adapter that parses its tool calls, the cache that serves its prompts, the verifier that grades its answers. Each shows a number you rely on turning out to be a property of that machinery, and badly wrong while looking perfectly stable.

Today's pick

0.00 vs 0.96

the same model's tool-call score — one serving adapter versus another, everything else held fixed

Interface-Induced Trajectory Censoring

An agent evaluation reports how often the model called a tool, and that number is read off the serving stack — the layer that formats your prompt and parses the reply. This paper shows the number can sit at zero while the model is emitting perfectly well-formed calls: the chat template and the output parser are two halves of one contract, and when they disagree the server discards good calls before anything downstream sees them. Holding weights, cases, decoding and seeds fixed and changing only that adapter, the same model scores 0.00 or 0.96, and a two-by-two over template and parser puts both main effects at exactly zero — no single component is defective, and repairing one side alone buys nothing. On a 115-task interactive retail agent benchmark the same swap moved server-parsed calls from 0 to 636; run something like the author's released 98-line preflight check before you trust any tool-call number, your own or a vendor's.

open the explainer → abstract PDF code

36% vs 75%

agent runs whose trajectory changed once the prompt cache was on — 16-bit weights versus 4-bit

Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

Prefix caching — the serving engine reusing the stored attention state of a shared prompt across requests — is on by default in the major open-source stacks and treated as a transparent speedup. Holding the model, decoding parameters, seed and request order fixed and issuing every request serially, this paper ran an eighty-episode multi-turn agent workload with the cache on and off: turning it on changed the agent's trajectory on 36.2% of episodes at 16-bit precision and 75.0% at four-bit, while with it off repeated execution was bit-identical in all 800 episodes. The cause is pinned down rather than guessed at — cached serving is deterministic given the cache's contents, but those contents are not part of the request and are never reset by default, so a rerun quietly starts somewhere else. If a self-hosted quantized model gives you an eval you cannot reproduce, this names the knob.

open the explainer → abstract PDF code

3% vs 32%

error the cascade's own dashboard reported, versus the error it actually delivered

Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades

A cost-saving cascade answers most queries with a cheap model and escalates the hard ones to an expensive model acting as verifier — and because the verifier decides what counts as wrong, it also computes your quality metrics. The share of the cheap model's wrong answers the verifier waves through grows as the cheap model gets better, from 0.12 at 0.5B to around 0.55 by 14B before it plateaus, because a stronger student's mistakes are simply more convincing — so the blind spot is worst in exactly the cheap-on-cheap regime cascades exist to create. A frontier verifier cuts that to about 0.05 but hands back the saving, escalating 46% of hard maths queries against a true error rate of 39%; fine-tuning the cheap model on the rejected tail degraded and eventually collapsed it under every teacher tried. Meanwhile the dashboard reads a flat 3% error while delivered error swings up to 32%: any metric computed through the verifier is blind to the verifier's own misses, so audit a sample against ground truth the cascade never sees.

open the explainer → abstract PDF code