Soft context compression replaces a retrieved document with a handful of memory tokens the frozen model reads instead of the text. The ratio is fixed when the compressor is trained and then applied to everything, so a dense contract and a boilerplate FAQ get the same budget, and every ratio you want to serve is another model to train. Draw the budget at random per example during fine-tuning and one model works at any ratio; then let a two-layer head read the document and pick its budget in a single pass. At serving batch sizes that halves the context cache and adds 47% to decoding throughput for under 0.7 points of answer quality, at 158–236× average compression.
A soft compressor takes a passage and encodes it into K memory tokens — continuous vectors, not words — which a frozen language model then consumes in place of the original text. Fewer memory tokens means a shorter prompt and a smaller cache; it also means less of the passage survives. The paper works with three budgets: 34 tokens, 10, and 1.
The constraint that matters is that K is baked in when the compressor is trained. If you want to serve three ratios you train three models, and whichever one you deploy applies its ratio to every document regardless of what that document contains. Step through the budgets, and through the three compressor families the paper tests, to see what each one costs.
The obvious reading is that quality falls monotonically as the budget shrinks, and on average it does. But the averages hide something the paper is careful to report: on 11.8% to 12.8% of in-domain samples the single-token budget beat the 34-token one, because the extra room preserved misleading surface detail the model then latched onto. More context is not reliably better context, which is the whole argument for choosing per document rather than globally.
The training change is one line of sampling. Instead of fixing K, each training example draws its own budget uniformly from the supported set, the encoder compresses into that many slots, and the frozen decoder is supervised on the answer as usual. One model ends up able to operate at any of the budgets.
The result that makes this worth doing is that it costs almost nothing. Compare a single sampled-budget model against three separately trained specialists, budget by budget.
Worst case across nine comparisons is about 1.2 F1 given up; several go the other way, and one — the tightest budget on the second compressor — gains 3.7 points in domain and 3.1 out of it, because a model that has also seen roomier budgets learns a better single-token summary than one that has only ever had one slot.
One caveat is load-bearing and easy to get wrong: the sampling belongs in fine-tuning only. Applying it during pretraining as well cost 2.3 F1 in domain and 2.5 out of it on the strongest compressor. The foundation has to be learned at a stable budget before the model can be taught to stretch.
Having one model that works at every ratio is only useful if something decides the ratio. The paper offers two mechanisms with very different cost profiles.
The first is a cascade. Run the smallest budget, read the decoder's confidence in the answer it produced — the length-normalised log-probability, which correlates with answer quality at about 0.57 — and accept it if it clears a threshold; otherwise escalate to the next budget up. The largest budget always accepts, so the ladder terminates. Drag the confidence and pick an operating point.
The cascade is accurate and it is not free: escalating means encoding and decoding again, so a document that goes the full distance has paid for three passes. That rules it out where a single query's latency is what you are optimising.
The second mechanism removes that cost. Push the document through the encoder without allocating any memory slots, mean-pool the final-layer hidden states into one vector, and run a two-layer network over it that predicts which budget this document needs. On a 1B model that head is 2.1 million parameters — 0.17% overhead — and the prediction ignores the question entirely, which means it can be computed once per document and cached at index time.
The reason to care about any of this is memory. At batch sizes that look like serving — 96 concurrent requests over an eight-thousand-token context on a 1B model — the context key-value cache is the binding constraint, and the budget decides its size directly. Pick a metric.
The single-token budget is the fastest row in the table and the least accurate; the 34-token budget is the most accurate and the slowest. The predictor lands between them on every axis and — this is the point — it does so per document rather than as a global compromise, which is why it holds answer quality within 0.7 points of the roomiest setting while giving back half the cache.
Set how many documents you put in front of the model per request and how long they run, then compare a fixed global budget against a predicted one. The per-budget cache figures are the paper's measurements at batch 96; the volume and document length are yours.
The move this argues for is small and specific. If you already truncate retrieved context to a fixed length, that length is a global guess about documents you have already indexed — and you can replace it with a per-document decision that costs one cached forward pass at index time.