Ten judges, three and a half opinions

Three papers on the gap between how much you added and how much it bought: a panel of ten judges worth three and a half, a four-tool interface that beats both eight tools and one, and eleven of twelve memory systems losing to no memory at all.

Today's pick

10 → 3.5

judges in the panel, and the number of independent judges their agreement is actually worth

Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus

Running every answer past ten judge models and taking the majority feels like ten opinions, and the statistics computed on the result assume exactly that. Across a bank of ten open-weight judges the errors correlated at 0.21 on average — they were wrong on the same items — which leaves the panel carrying about as much independent evidence as 3.5 judges; three frontier judges from three different providers were worse at 0.56, or 1.4 judges' worth. Pooling every vote as a separate observation understates the variance by 2.85 times, and in 28% of head-to-head comparisons a system that looked significantly better stopped being significantly better once the shared errors were counted. Score about a hundred trusted examples first, measure how often your judges fail together, and choose the voting rule there rather than on the data you are about to make a decision from.

open the explainer → abstract PDF

+16.4%

more tasks completed with four mid-grain tools instead of eight to ten single-purpose ones

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents

How you carve a tool server into tools is usually whatever the underlying API happened to look like, and this study makes it the variable instead: 81 multi-step scenarios across 9 domains, 9 locally run models from 268M to 20.9B, 8,748 trials, each run at four interface shapes — eight to ten single-purpose tools, four grouped by function, two split into read and write, or one tool for everything. Four tools completed 16.4% more tasks than the fine-grained set and 33.6% more than the single tool, and doubled the share of calls with correct arguments, from 0.20 to 0.40. The consolidated end fails in a way worth recognising: 28.2% of runs against one monolithic tool called no tool at all, against 10.2% at the fine-grained end, the model either announcing it was done or reaching for a tool name it remembered. Model size barely tracked completion (correlation 0.285, not significant) while it tracked latency almost exactly (0.946), so regrouping a sprawling registry is the cheaper move than upgrading the model.

open the explainer → abstract PDF

11 of 12

pairings of a coding agent with an off-the-shelf memory system that failed to beat the same agent with memory switched off

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

A memory layer on a coding agent is supposed to let it reuse what it worked out on the last ticket, and on 111 real repository tasks the experience itself was genuinely worth having: hand it to the agent directly and resolution rose by 1.1 to 4.5 points on four of five held-out models, with every one of them finishing in fewer steps. Ask four existing memory systems to construct and retrieve that same experience from the same history and eleven of twelve pairings landed below the memory-off baseline, several by 4 to 5.5 points. The diagnosis is the useful part: only 16.0% of the failures were the system never storing the right history and 1.3% were retrieving the wrong thing, while 69.3% stored it in a form that had lost the fix — systems that keep raw transcripts carried stale instructions into the new task, and systems that distil into tidy facts generalised the specific change away. Before you add a memory framework, run one known-useful piece of experience through it and check that what comes out the other side still contains the fix.

open the explainer → abstract PDF