Your retriever already knows

Three papers on the signals you read before you trust an answer: the shape of your own retrieval scores, the confidence a judge states out loud, and what ordinary typos do to the verdict.

Today's pick

0.856 vs 0.649

separating sufficient from insufficient retrieval, score shape against a model that reads the retrieved pages

Your Retriever Already Knows: Distribution-Shape QPP for RAG Retrieval Sufficiency

A retrieval pipeline usually has no way to tell, at the moment it answers, whether the pages it just pulled actually contain the answer — so on an out-of-scope question it invents one. This work throws the document text away and reads only the shape of the similarity scores the retriever already produced: how far the top hit sits above the tenth, how fast the curve decays, where it elbows. Twenty-four such features fed to a small classifier separate sufficient from insufficient retrieval at 0.856 area under the ROC curve across 14,514 queries in eight domains, against 0.649 for a 35-billion-parameter model that reads the retrieved pages and 0.593 for the top score on its own — at 2 milliseconds a query instead of 6.3 seconds. Train it on your own logged retrievals and gate generation on it, but budget for the drop the authors measure on a domain it has never seen: 0.856 falls to 0.706.

open the explainer → abstract PDF

+0.6 pts vs −1.1 pts

balanced-accuracy change when a calibration prompt is added, on a post-2025 flagship against a pre-2025 one

Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models

Asking a judge to state a confidence number was long the weak option for getting a graded score — overconfident, clustered on round numbers — with token log-probabilities the respectable alternative. Across three judging datasets and up to eighteen models that ordering has flipped for models released after 2025: adding an overconfidence warning plus a three-step self-debate prompt — argue true, argue false, then conclude — cuts calibration error from 8.9% to 3.8% and widens the usable score range from 75.8% to 89.6% on one flagship. The same two additions cost GPT-4o 1.1 points of balanced accuracy while GPT-5.2 gains 0.6, so this is a property of the generation you judge with rather than a universally good prompt. It matters now because log-probabilities are vanishing from the APIs anyway — Claude never returned them, and Gemini dropped them at 3.1 Pro.

open the explainer → abstract PDF

240 vs 2

neutral answers a judge relabelled as biased after ordinary typos, against biased ones it relabelled neutral

When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text

If you measure social bias by having one model judge another model's answers, the text being judged is rarely clean — real users write with typos, dropped punctuation and informal spelling. Applying five kinds of surface noise at realistic strength to 3,822 answers, the authors find the damage runs almost entirely one way: the weakest judge turned 240 neutral verdicts into biased ones and only 2 biased verdicts into neutral, a 120-fold imbalance, and stacking noise types pushed that to 288 against zero. Stronger judges shrink the gap rather than reversing it — GPT-5.4 sits at 30 against 24 — and the distortion is purest at mild, realistic corruption, where fabrications pile up before any erasure starts. Normalise or spell-correct text before a judge scores it for bias, and read any bias rate measured on raw user writing as an overestimate, largest in the categories you would least want to get wrong.

open the explainer → abstract PDF code