Hybrid search wins the average, loses the query

Three papers on changes that look fine in aggregate and land badly one case at a time: the second retriever that lifts the average while demoting the answer on up to half of queries, the deletion that removes the record but not the answer, and the extra context that fixes one prediction by breaking another.

Today's pick

13-48%

of queries where blending a keyword index into vector search ranked the right document lower than the better of the two retrievers would have alone

The Fusion Tax: A Diagnostic Analysis of When Hybrid Retrieval Hurts

Hybrid retrieval — a keyword index such as BM25 blended with an embedding model, by reciprocal rank fusion or a weighted score mix — is the standard advice, and on averages it earns it: the best blend beat the embedding retriever alone on 33 of 35 encoder-and-corpus pairs. Scored query by query across seven retrieval benchmarks, five embedding models from 22M to 596M parameters and three blending functions, that same blend put the relevant document below whichever retriever was better on its own for 13-48% of queries and below the plain embedding retriever for 8-21%, while the average it bought was 0.008 to 0.051 on a 0-to-1 top-ten ranking score. Which queries it hurts is close to unguessable: a classifier over 17 query features managed 0.59 average accuracy-under-curve on a corpus it had not seen, so a per-query router never beat a fixed blend. Their closing rule is the usable part — label 200 to 400 of your own queries, score the keyword and embedding retrievers separately, and only blend when the gap between them is small.

open the explainer → abstract PDF

62-99%

of cases where an agent rebuilt a deleted value exactly, from the labelled pieces a summary had kept after the original record was gone

AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents

An agent's memory is not just the rows you wrote; it is the summaries and extracted fields built from them, so deleting the row you addressed leaves its descendants in place. The audit pairs an agent that saw a secret and then had it deleted against a matched agent that never saw it: one plain question — what is my attribute — pulled the deleted value back out of a 7B open model in 68% of 200 profiles, against 0 of 200 for the never-exposed twin. It survives having no copy at all: when the summary held three labelled pieces plus the public rule for joining them, and no contiguous copy of the value appeared anywhere the model could see (0 of 1,248 states), exact reconstruction ran 62-99% across three model families. The fix that worked was retiring or rebuilding every derived record whose recorded sources included the deleted row, which took recovery to zero, while a filter blocking the exact value from being written back stopped re-storage but left fresh-session recall at 64-68%.

open the explainer → abstract PDF

A Reality Check of Inference-Time Context Augmentation Methods

Inserting text before the model answers — blank filler, a verbatim repeat, a paraphrase, the model's own elaboration, or a larger model's explanation — is judged by whether accuracy went up, which hides that every insertion rescues some answers and breaks others. Across seven models from 1B to 72B, three question sets and insertions placed one sentence at a time, a verbatim repeat moved one 8B model's medical-exam accuracy from 53.6% to 43.2% (3.9% of positions rescued, 14.4% broken), while a larger model's explanation took a 4B model from 45.7% to 55.0% (+14.1 rescued, -4.7 broken). Their own check is the thing to copy: re-run the per-question score shifts with random signs a thousand times, then keep the method only if its rescues beat that noise while its breaks do not — on which test the strongest method passed just 8 of 21 model-and-dataset pairs, and the insertions that add no new information mostly flipped answers the model was already unsure about. Count rescues and breaks per item before you keep a trick: a flat accuracy line can hide 3% churn each way or ten points of both.

open the explainer → abstract PDF