The loop beats the prompt

Three papers on knowing whether the output is right: one measures the draft after generation and edits until it lands, one reads the reasoning trace while it happens, and one tries to score the prompt before a token is written.

Today's pick

21-32% → 93-99%

share of drafts landing inside the target reading-level band, single-shot prompting versus a measure-and-edit loop

A Closed-Loop Control Architecture for Reliable Constraint Satisfaction in LLM Text Generation

Product features that have to hit a number — a reading level, a word count, a tone score — are usually built by putting the number in the prompt, and across four commercial models that landed inside the target band only 21% to 32% of the time. This paper wraps the model in an ordinary control loop instead: generate, score the draft with deterministic code, hand the model one diagnosed edit, re-score, stop when it lands — which hit the same targets 93% to 99% of the time in 1.7 edit rounds on average. The model is never asked to judge whether it met the constraint; it only writes and edits, and the measurement stays in code where it is reproducible. Their content gate checks recall only, so an edit that reverses “increased 20%” to “decreased 20%” still passes — copy the loop, but add a factual check when the text carries claims.

open the explainer → abstract PDF

+27.5 pts

more correct answers on competition maths when the model's step-by-step uncertainty fell at every step instead of rising somewhere

Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction

If you could tell which answers to distrust from the reasoning trace itself, you could route only those to a second check. This independent reproduction, preregistered before the runs, confirms a narrow version of that: what predicts correctness is the shape of the model's step-by-step uncertainty — whether it falls at every step — and not how far it fell in total. Chains whose uncertainty never rose were 9.6 points more likely to be right on grade-school word problems and 27.5 points more likely on competition maths, while the size of the total drop correlated with correctness at −0.02 on the first benchmark, which is to say not at all. The cheapest finding is the useful one: ranking answers by the uncertainty at the final step alone beat the all-steps monotonicity flag in every model-and-benchmark cell tested, so you can resample the last step a few times rather than instrument the whole chain.

open the explainer → abstract PDF

7.6 pts → 2.1 pts

the pass-rate gap across the reported complexity threshold, before and after controlling for what kind of task the prompt asked for

The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability

Most teams score how hard a coding task was by looking at the code that came back, and this paper's opening move is to call that broken: a prompt hard enough to make the model fail often yields a short, simple-looking program, so the failure drags the difficulty score down with it. Their alternative is a six-dimension rubric — branching, iteration, state, data structures, edge cases, composition, each scored 0 to 4 — read off the prompt before generation, which four judges agreed on closely and which barely moved under paraphrase or translation into Java and C++. The headline finding is worth reading as a warning about your own dashboards: pooled over 5,000 prompts and 21 models, pass rates sat 7.6 points higher above a composite score of 13.75 than below it, but controlling for what kind of task the prompt asked for shrank that gap to 2.1 points and moved the threshold to 10.75, and controlling for which batch the prompts came from moved it to 8.50. Use the rubric as a pre-generation routing signal; treat any threshold you find in your own data as a fact about your prompt mix, not about the models.

open the explainer → abstract PDF