You change a tool description or a line of agent instructions and you want to know whether it helped. The full suite costs too much to run on every change, so you run a handful of tasks instead — and a handful of tasks, chosen by eye, is mostly noise. Resampling a published set of coding-agent trials showed that only 22 of 113 tasks tracked full-benchmark performance reliably enough to read from a single run. The method here finds those tasks, puts their partial results on one scale, and buys as many as your budget allows.
Start from a published set of trials where every task was run four times. Draw one of those four runs per task, at random, and correlate that single-run picture against the full-benchmark score across the model and effort configurations available. Do it ten thousand times, and rank each task by the fifth percentile of its correlation — the bad-luck case, not the average one. A task ranks high only when almost every combination of single runs still reproduces the ordering the full benchmark gives.
Only 22 of the 113 tasks cleared a fifth-percentile correlation of 0.50. Drag the size of a set picked without that ranking.
There is a second loss hiding in the usual setup: pass-or-fail scoring throws away everything a run did achieve. These trials record the fraction of previously failing tests that a run turned green, so a run that fixes three checks out of five is distinguishable from one that fixes none. The method keeps that fraction and uses it as the measurement.
Four steps, in order, and the order matters: ranking never changes after prices are considered. Step through them.
The budget step is the one that makes this usable day to day. Set a per-comparison budget and the scan walks the frozen ranking from the top, taking every task that still fits and skipping the ones that do not. Drag it.
Benchmarks publish results from a neutral reference harness. You ship a different one: your own scaffolding, your own tool set, your own instructions. On the eight selected tasks, the published estimate for this model at low reasoning was −0.54%. Run the same eight tasks through the author's actual harness and the baseline is 20.43% — a 21.0-point gap, with the published figure also assuming a cost 13× lower than what the run actually recorded.
Toggle the harness.
This is not a quirk of one setup. Holding the model and the reasoning effort fixed and changing only the harness moved a different model's score by 5.1 points on a terminal benchmark. Published numbers rank models under someone else's scaffolding; they cannot baseline yours. Re-measure the selected tasks where you actually run them, and do it before the first comparison, not after a surprising result.
With eight tasks frozen and re-baselined, the author iterated on the agent's own skills and instructions: dispatch fixes, a resume-until-complete change, scope checks. Each evaluation is a baseline-versus-candidate comparison on the same frozen set. Step through the recorded ones.
The honest reading of the endpoint is two different statements. The cost reduction — $4.18 to $1.75, 58.1% less — came out significant, and all eight matched tasks favoured the adopted version. The score improvement, 36.46% to 42.36%, did not: with the variance transferred from published runs it lands at p=0.326. The decision the data supports is “adopt, it is much cheaper and no worse”, not “adopt, it is six points better”.
A cheap eval that reports which of its two numbers you are allowed to believe is worth more than an expensive one that reports both with equal confidence.
You have an agent feature and a queue of changes to it: a rewritten tool description, a retry policy, a skill file. Each change needs a baseline-versus-candidate comparison you trust enough to act on. The question is how many of those you can afford per month, because that number is what actually sets your iteration speed.
The trade is explicit and worth saying out loud to whoever reads your numbers: you have bought iteration speed on one system under one specification, and given up any claim about unseen tasks. Change the task set and you have a different benchmark whose results cannot be pooled with last month's. Freeze it, write down what it does not cover, and keep the full suite for the decisions that deserve it.