When one model grades another's output, you usually want a graded score rather than a yes or no — something to sort by, threshold, and put a confidence interval on. There have been two ways to get one: read the model's token probabilities and compute an expected rating, or simply ask it how confident it is. The second was the discredited option — overconfident, clustered on round numbers, jammed into a narrow band. Across three judging datasets and up to 18 models, that ordering has flipped for models released after 2025. Adding an overconfidence warning and a three-step self-debate improves calibration error from 8.9% to 3.8% and widens the usable score range from 75.8% to 89.6% — and where a pre-2025 model pays 1.1 points of accuracy for it, a post-2025 one gains 0.6.
A binary verdict is not much use in an eval harness. You want a number: to rank borderline cases, to set an operating threshold, to see whether a change moved anything. There are exactly two places that number can come from, and they are different things wearing the same clothes.
Switch between them.
The practical half of the argument is availability, and it has moved underneath everyone. Log-probabilities require the provider to return them, and providers have been withdrawing them: the GPT family is the only frontier line that still exposes them across several releases, Claude never did, and Gemini dropped them at 3.1 Pro. That alone would make the comparison worth redoing. The paper's finding is that by the time you are forced to switch, the replacement has become the better of the two.
The old objection to asking a model for a confidence number was not that it lies. It was that it uses almost none of the scale: everything lands on 80, 90 or 95, so the thing you meant to sort by has three distinct values in it. The paper measures that directly as spread — how much of the available range the scores actually occupy, against a flat distribution. Below, drag how much of the scale the judge uses and watch what a downstream threshold has left to work with.
Two prompt additions fix most of it, and the paper separates their contributions. Step through them.
The advisory is the calibration lever: on its own it reduces calibration error on all ten models in the cohort and supplies most of the widening — 18.9 points of spread on pre-2025 models and 11.6 on post-2025 ones. Self-debate barely adds to the spread, but it is what stops free-form reasoning from eroding it, because unconstrained reasoning tends to build one argument and then agree with itself. On one flagship the pair together takes calibration error from 8.9% to 3.8% and spread from 75.8% to 89.6%.
This is the finding that makes the paper more than a prompting tip. The advisory and the self-debate are not free — they ask the model to reconsider, and a model that reconsiders badly ends up worse. Whether they are free turns out to depend almost entirely on when the model was released.
Named examples make the size of it concrete: adding both to GPT-4o moves balanced accuracy from 81.3% to 80.2%, and adding both to GPT-5.2 moves it from 82.4% to 83.0%. A point either way is not dramatic on its own — what matters is the sign, because it decides whether calibration costs you accuracy or comes free.
Two other diagnostics move the same way. The gap between what the judge predicts and the best score an oracle could have extracted from its confidence shrinks toward zero on post-2025 flagships, and so does the accuracy drop caused by making the model argue both sides. On newer models the stated number and the verdict are the same object; on older ones the number was a decoration on top of a decision already made.
Judging faithfulness is comparatively easy: humans agree with each other, so a judge that tracks them looks good. Judging whether a summary is relevant is not — on the three dimensions the paper uses, human agreement runs from 0.80 down to 0.40, and the usual result is that judge quality falls with it. That decline is the thing eval harnesses quietly live with.
The measured quantity is the slope of judge quality against human agreement, and the paper's protocol has a significantly flatter one — 0.143 flatter, one-sided p below 10⁻⁴. The log-probability signals go the other way: weaker regression fits and smaller slopes, which sounds like the same robustness but is not. A flat line with a good fit means the method holds up on hard tasks; a flat line with a poor fit means the signal was not tracking anything much to begin with.
Pick the judge you actually run and the protocol you would put in front of it, and read off what the paper's measured deltas do to a review queue. The per-model figures are the paper's; the queue arithmetic on top of them is illustrative.
The move to make is small and testable: put the advisory and the self-debate behind a flag, run your existing eval set both ways on the judge you already use, and keep whichever wins on your own data. If your judge predates 2025 you should expect to lose about a point of accuracy for the calibration, and the right answer may be to upgrade the judge rather than the prompt. If it does not, you get the calibration for nothing — and you stop depending on a token distribution that two of the three frontier families no longer return.