The same query came back different

Three papers about measurements that will not hold still: the pipeline that answers differently on a rerun, the agent's test suite that passes without catching anything, and the rubric that only taught the judges to agree with each other.

Today's pick

4.0% → 67%

share of extracted values that changed between identical runs, and the share of query answers that changed with them

Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

Set temperature to zero and the same request still comes back different, because inference servers batch concurrent requests and the batch yours lands in changes the floating-point arithmetic behind the logits. Replaying three benchmarks of database queries whose predicates are evaluated by a model — five runs each, 1.2 million calls, every input checksummed identical — between 0.07% and 10.2% of rows changed the verdict their operator gave, 24.0% on a 24-way classification, and the rate climbs with every extra run rather than settling. What compounds it is the query plan: 4.0% of extracted values changing moved the result table of 83 of 123 queries, because a query reading one field from every document needs only one field to move, and the paper gives the law — an answer that exposes n rows reproduces about e^(−n·δ̄) of the time, so one expected disagreement already cuts reproduction to a third. Voting three times doubles your calls, removes 36% of the disagreement and makes none of the aggregate or ranking answers reproducible; recording the per-row outputs an answer came from, at roughly 39 bytes each, is the fix that works.

open the explainer → abstract PDF code

96% vs 56%

share of a coding agent's own tests that pass on the correct build, against the share of planted requirement violations the same suite catches

TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications

A coding agent writes a test suite from your specification, every test passes, and the suite does not detect the faults it was meant to. Across 165 command-line applications, each with a written specification and about 74 planted faults that violate exactly one stated requirement apiece, the best agent's suite was 96.1% valid — its tests pass on the correct build — while catching 55.5% of the faults, and four of six agents stayed under 40%; the reference suite catches all of them, so the ceiling is real. The cause is input design, not an ambiguous spec: handing the agent the working binary to observe lifted validity to 98.4% and detection by 0.4 points, and 5.3× more spend on reasoning bought 15 points and still finished under half. The same models wrote implementations passing 82% of reference tests on average while their suites caught 38% of faults, and 41% of the faults were missed by every agent tested, so a green suite the agent wrote itself is close to no evidence at all.

open the explainer → abstract PDF code

+0.069 vs −0.007

change in how much five judges agreed with each other, against the change in how much they agreed with the human experts

Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans

When a panel of judges splits on a score, the standard reading is that the rubric is ambiguous — so this pre-registered study cut each rubric into clauses, ranked them by how often five judges disagreed, and had an editor model rewrite only the most contested ones. It worked in the narrow sense: on a summary-quality benchmark agreement among the judges rose 0.069, and 0.074 above a matched control that applied the same edits to the same number of randomly chosen clauses, so the targeting rather than the editing did the work. Agreement with the three experts' ratings moved −0.007, and in point estimate that random-clause control finished closer to the humans on all three datasets. A label-free edit has no way to know which reading the annotators held, so it commits the panel to one that is self-consistent and may be wrong: use disagreement to find the clause, then a handful of human labels to settle it, and stop reading inter-judge agreement as rubric quality.

open the explainer → abstract PDF