Almost every difficulty dashboard for code generation scores the output: lines, branches, cyclomatic complexity of whatever the model produced. That measurement is contaminated by its own outcome. A prompt hard enough to make the model give up yields a short, simple-looking program, so the failure drags the difficulty score down and the hardest tasks get filed as easy. This paper scores the prompt instead, on six dimensions, before a token is generated — then uses 5,000 prompts and 105,000 samples from 21 models to show both what that buys and how fragile the headline it produces turns out to be.
The paper's opening sentence is the whole argument: complexity measured from generated code is failure-dependent. The measurement and the thing it is supposed to predict share a cause, so the number is not a difficulty score — it is a difficulty score with the answer mixed in.
Step through three requests and watch the two measurements disagree.
The examples are drawn for the explainer; the mechanism is not. Once the measurement moves with the outcome, every downstream number inherits the contamination — your per-bucket pass rates, your “model X handles hard tasks” claim, your decision about which prompts to route to a bigger model.
The alternative is a rubric applied to the request itself. Six dimensions, each scored 0 to 4, summing to a composite between 0 and 24. Four language models outside the evaluated panel scored all 5,000 prompts, agreeing at an intraclass correlation of 0.872 across nearly 20,000 score rows — high enough that the index is a usable instrument even where the conclusions drawn from it are not.
Score a request yourself.
Two robustness results matter more than they sound. Rewriting a prompt without changing what it asks for moves the composite by 0.115 points on average, with a rank correlation of 0.963. Translating the task to Java or C++ keeps it at 0.992 and 0.969. The index measures the request, not the phrasing and not the language — which is what you need from something you plan to route on.
With the index in hand, the obvious question is whether pass rate falls as the composite rises. It does not, at least not monotonically. Pooled over 5,000 prompts and 21 models, the pass rate declines through the middle of the index, troughs near a composite of 10, and rebounds in the denser high region. A breakpoint search puts the change at 13.75, with 79.9% passing below it and 87.6% above — a 7.6-point gap in the direction nobody expects.
Move the cursor across the index.
Before reading anything into the rebound, note what sits underneath it: only 14 audit-clean prompts scored above a composite of 16, even after the authors extended the sample by 365 prompts to go looking for them. The top of this index is nearly empty, and a rebound measured in a nearly empty region is a hypothesis, not a finding.
This is where the paper earns its keep, because it goes looking for its own headline's weak points instead of defending them. Add fixed effects for the nine kinds of task the prompts ask for — string manipulation, data structures and algorithms, and so on — and the breakpoint slides from 13.75 to 10.75 while the regime gap collapses from 7.6 points to 2.1. Control instead for which of two construction batches a prompt came from, and it moves to 8.50.
Step through the specifications.
An estimate that moves this much under ordinary controls is telling you about the sample. The authors say so themselves, and go further: their overidentification tests reject the joint instrument restrictions across the six rubric dimensions, and they present the whole thing as a measurement framework rather than a causal claim. The honest reading is that the rubric replicates and the kink does not — which is still a useful day's work, as long as nobody turns 13.75 into a config value.
Here is the failure mode to take home, using the paper's own two prompt batches. One batch averages a composite of 7.92 and passes 74.7% of the time. The other averages 11.42 and passes 88.0%. Neither batch on its own reproduces the pooled 7.6-point regime gap; the gap is what appears when you pool them.
Slide the mix of your eval set and watch a complexity “finding” assemble itself out of composition alone.
Nothing about any model changed as you moved that slider. Both numbers moved together because the two batches differ in both, and a dashboard that reports pass rate against complexity will draw you a relationship out of that. The defence is boring and effective: fix the eval set, record its composite distribution alongside the pass rate, and compare like with like when the set changes.