A number for how likely this answer is right decides what you ship, what you escalate and what you retry. The default way to get one is to sample the model ten times and see how much it agrees with itself — ten generations, and it degrades exactly where agents live. Ask instead what happened the last 50 times this model met a task like this, feeling this sure, and you match or beat voting on 23 of 24 model-and-benchmark pairings at a tenth of the generation cost.
Two questions decide whether a confidence score is worth anything. Does it rank? — if you sort answers by it, do the wrong ones sink to the bottom. And does it mean anything? — when it says 0.8, is it right about four times in five, so you can set a threshold once and leave it. The first is the ranking score below; the second is calibration error, where lower is better.
Two baselines dominate practice: ask the model how sure it is, or sample it ten times and measure the agreement. Step through the benchmarks and watch where each falls over.
Notice the shape. On multiple-choice reasoning, voting is respectable and the gap is small. On code and on agent work — where a rollout costs real money, where two rollouts are not comparable to each other, and where there is nothing to vote on — voting collapses or is replaced by a second model reading the transcript, and both trail badly.
Every completed attempt is filed as an episode: the task, the rollout, the answer, a short self-reflection written before anyone graded it, the confidence the model stated before it knew the outcome, the eventual pass or fail, and a one-line lesson written afterwards. The bank is that pile.
To score a new attempt, retrieve its 50 nearest neighbours and report what fraction of them were right. Everything turns on what “nearest” means — and the surprise is that the task text alone is not enough.
Adding the model's own stated confidence to the key is worth about .08 of ranking power on reasoning and .12 on agent tasks. That is the whole trick in one line: the stated confidence is nearly useless as a number, and quite informative as a coordinate. “Times I felt this sure about a task like this” is a much sharper reference class than “times I saw a task like this”.
Then the second reading. The model is handed those neighbours as short cards — task, how sure it felt, how it went, what it learned — and asked to name any recurring failure mode the record reveals before restating its confidence. The final score is the plain average of the retrieved success rate and that restated number. No training, no extra rollouts: one answer plus one short call.
This is the part that decides whether you can use any of it. The method is a thin wrapper around a pile of graded outcomes, so the questions that matter are: how big does the pile need to be, whose pile can it be, and how good do the grades have to be. Step through the answers.
Size matters more than any of it. On an app-using agent benchmark the ranking score with a hundred episodes was .628 — barely worth the call — and .810 on the full pool. Where a curve flattens it is because the pile is already dense; the benchmarks with the fewest episodes were still climbing when the paper ran out of data. And one boundary is hard: outcomes graded by the model itself scored worse than a bank with no outcome labels in it at all. A weak but independent grader, agreeing with the gold labels 91% of the time, kept most of the value.
A confidence score earns its keep at one moment: when you hold something back. Hold back the least-confident tenth and hand it to a human, a bigger model or a retry, and the success rate of everything you did deliver goes up. Toggle the hold-back.
The decile table is the reason it works. Sorted by this score, the most confident tenth of attempts is right 98% of the time and the least confident tenth 28%, and the bottom fifth of the ordering holds half of all the errors in the run. That is a working triage rule, not a summary statistic.
Suppose a thousand agent runs a week at the accuracy you actually see, and a budget for sending some fraction to a human or a stronger model. The question is how much of your error you can catch for that budget. Set both and watch.
The shape is what transfers, not the arithmetic. A score that concentrates errors in the bottom of the ordering turns a fixed review budget into a large share of your failures caught; a score that does not is a random sample with extra steps. Which is the thing to measure before you build any of this: sort last month's graded runs by whatever confidence you already collect, and see what fraction of your errors sits in the bottom fifth.