A retrieval pipeline answers the same way whether it found the right page or nothing at all. This paper asks whether the retriever has already told you which it was — not in the documents it returned, but in the shape of the similarity scores attached to them: how far the top hit sits above the tenth, how steeply the curve decays, where it elbows. Twenty-four such numbers, computed from the score vector and nothing else, separate sufficient from insufficient retrieval at 0.856 area under the ROC curve across 14,514 queries in eight domains — against 0.649 for a 35-billion-parameter model that reads the retrieved pages, and 0.593 for the top score alone. It costs 2.0 ms a query instead of 6.3 s.
Retrieve, stuff the top pages into the prompt, generate. On a question the corpus cannot answer, every step of that runs exactly as it does on a question it can — the retriever always returns k pages, the model always writes something, and the invented answer looks the same as the grounded one. The usual fix is to bolt a second model on the end and have it read the pages: reliable in principle, slow and expensive in practice.
The paper's question is whether you need to read anything at all. Switch between the two ways of asking.
The labels underneath are blunt: a query is sufficient if a page rated fully relevant — one that answers the question on its own — appears in the top k. Everything else is insufficient, and it is the group that produces confident nonsense downstream. On the regulatory corpus the authors deliberately included 500 adversarial unanswerable queries, and separating those from the answerable ones is where the score shape does its best work of all, at 0.934.
The retriever hands back a vector of similarity scores. Almost every pipeline reads exactly one entry of it — the top score — and throws the rest away. The paper's claim is that the discarded part carries most of the signal, because a query the corpus can answer produces a different curve than one it cannot: one page towers over the rest, then a cliff. A query with no answer produces a gentle slope of near-misses, all about equally mediocre.
Drag the cliff and watch the features move. The curve is illustrative; the feature definitions are the paper's.
Step through the three families the twenty-four features fall into. Only one of them is about the query at all, and none of them opens a document.
Features go through a standardiser into a small binary classifier — logistic regression, ridge, or a modest multi-layer network, whichever three-fold cross-validation prefers on that dataset — and out comes a number in zero to one. That is the whole apparatus. The interesting design decision is negative: the authors deliberately did not reuse classical retrieval-quality statistics wholesale, because those were built to predict ranking quality from text. Only one feature, the maximum similarity, overlaps with the classical set.
Nine predictors, one instrument, 2,905 held-out queries pooled across eight domains. Four read only the score vector, one reads the retrieved pages with a 35-billion-parameter model, and three are hybrids that do both. Switch the axis from accuracy to cost and the ranking inverts.
Read down the score-only column and the pattern is monotone in how much of the curve you look at: one feature gives 0.593, four gives 0.643, six gives 0.714, eighteen classical ones give 0.835, and the twenty-four shape features give 0.856. The gain over the classical set is small but it holds up statistically, and every gap below that is large. The model that actually reads the pages lands at 0.649 — better than the top score alone, worse than six hand-made statistics, and roughly three thousand times slower than either.
Adding that model on top of the shape features does help, and the paper is straight about how much: 0.856 to 0.863 on the pooled set. That is the price list. Seven thousandths of separating power for 6.3 seconds and a model call per query.
The second corpus is 517 pages of Czech nuclear-regulatory documents with 1,510 synthetic queries — standard, paraphrased, and deliberately unanswerable. It reproduces the ordering: same ranking of methods, Spearman 0.90 between the two datasets. It also shows why calibration is worth watching separately from ranking. The trimmed thirteen-feature model ranks a shade below the full one but is the best-calibrated thing in the table, and a gate is a threshold decision, so calibration is the property you actually operate on.
Every number so far is trained per domain: roughly a thousand to seventeen hundred labelled queries from the same corpus it is then tested on. That is cheap by machine-learning standards and awkward by product standards, because it means the predictor is not something you drop in front of an arbitrary corpus. Hold a domain out of training entirely and the cost shows up immediately.
Feature selection run under that hold-out picks thirteen of the twenty-four: eight distribution features and all five query features. Every corpus-level feature is dropped, which makes sense — a global similarity baseline is a property of the corpus you fitted on, so it is exactly the kind of thing that does not travel. The trimmed model gives up 0.074 in-domain to recover 0.013 across domains and ends up better calibrated than either. If you only ever serve one corpus, take the full model; if you serve many, the trimmed one is the honest default.
Across the eight domains the spread is 0.787 to 0.905, so this is not one easy domain carrying an average. The weakest is energy, the strongest computer science and physics — and even the weakest beats the page-reading model's pooled figure by more than a tenth.
Put a gate in front of generation: below the threshold, say you do not know or escalate; above it, answer. Pick the signal, set how aggressive the threshold is, and read off what it catches and what it costs. The separating power of each option is the paper's measured figure; turning that single number into a catch rate and a false-block rate assumes a textbook equal-variance curve, so the two rates below are illustrative shape, not measurements.
The shape of the trade is what to take away rather than any individual number. A gate built on the score curve is close to free, so you can afford to run it on every query and tune the threshold against real traffic; a gate that reads pages with a model costs a second-scale wait and a second bill on every request, and in this paper it separated the two cases worse. The signal was already in the response you had.