Every published study of the tool-server ecosystem draws its sample somewhere that quietly selects for servers that work — a reference list, a popularity ranking, a hand-picked frame, or a pipeline that repairs a server until it boots. This paper draws 400 servers at random from a census of 24,135, with a published seed, and probes each one exactly once with no credentials, no repair and no retry. 48.8% complete a handshake, against 66.7% for a hand-curated frame measured with the same instrument. The dominant failure is not a missing API key at 13.3% — it is servers that never start at all, at 37.5%. The same instrument then turns on the tool-use benchmarks everyone scores against, and finds that 68.8% of one corpus's rows are exact repeats.
The census is the easy part: sweep the official registry by cursor until it ends, twice, 39 days apart. The population grew from 16,548 servers to 24,135 over that window. Nothing in the census runs third-party code, so it scales to the whole published population.
The sampling is where every prior study quietly diverges. Reference sets, popularity lists and hand-curated frames all pick servers someone already knows about, and repair pipelines keep fixing a server until it starts — which means the servers that never start are removed from the denominator before anyone counts. This paper instead fixes a frame of the 7,258 servers that publish to npm, declare stdio and are marked active, sorts them canonically, and draws 400 without replacement using a seeded generator. The draw manifest records a hash of the exact frame bytes, so a redraw against a moved frame is detected rather than silently reported.
Switch the frame below and watch the headline move. Both sides are measured with the same probe.
The curated frame reports a start rate 17.9 points higher, and — because it has no failure taxonomy at all — it cannot tell you why a third of it did not run. The random draw can: of 400 servers, 195 completed a handshake, 150 never started, 53 stopped for want of credentials, and 2 packages were simply not there. Startup failure outnumbers credential-gating by almost three to one, which inverts the usual assumption that the integration problem is mostly secrets management.
The instrument speaks the wire protocol directly rather than going through a client library. For each drawn package it launches the server over standard input and output, performs the initialize handshake, enumerates the advertised tools, and validates each tool's schema against the constraints the specification places on tool definitions. Step through the rules that make the number mean something.
The discipline matters more than the tooling. A single attempt with no retry means a server that was merely having a bad minute is counted as a failure, so 48.8% is explicitly a lower bound on how many of these servers could start. Refusing to supply credentials means a server that legitimately needs a key is separated from one that is broken, instead of both landing in the same bucket. And refusing to repair anything is what keeps the denominator honest — every one of the 400 draws carries a recorded outcome, and none was dropped.
Among the 195 servers that did run, hard conformance is total. Across 2,766 advertised tools — 2,759 of them carrying a description — there are zero fatal schema violations, and zero servers with any. That replicates an earlier curated-sample result at roughly fourteen times the scale and, crucially, on a random draw, which is what turns it from a property of the sample into a property of the ecosystem: the required parts of the specification are enforced by the tooling everybody uses, so they are simply always right.
The optional parts are not. Four annotations exist to tell an agent whether a tool is safe to call before it calls it — whether the tool only reads, whether it destroys, whether it is idempotent, and whether it touches the open world. Curation flatters both headline numbers, and in opposite directions.
On the random draw, 1,626 of 2,766 tools carry none of the four annotations — 58.8%, against 41.5% on the curated frame. And the omission is not scattered tool by tool. Of the 194 servers advertising at least one tool, 72 annotate every tool they ship and 122 annotate none; there is no partially-annotated server anywhere in the sample.
That all-or-nothing shape is the useful finding, because it changes what you do about it. Annotation is not a field individual tool authors forget under time pressure; it is a decision taken once per server, by whoever set the project up. So a screening rule can be written at the server level — check one tool, and you know the rest — and a server that annotates nothing is telling you it will never tell you which of its tools writes.
The second half turns the same instrument on the corpora that tool-use scores are computed over. The method is held constant across all three: concatenate each tool's name and full description, deduplicate globally on that exact key first, vectorise per corpus, and count the share of surviving tools that have at least one neighbour above a cosine threshold.
Move the threshold. The solid rows count every pair; the paired rows count only pairs that span different authoring units — different servers for real tools, different tasks for the benchmarks.
Real tools show 2.8% near-duplication at the loosest threshold, and every point of it sits inside a single server — the list_x, get_x, create_x family that one author writes together. Across independent authors, near-duplication is 0.0% at every threshold tested. One benchmark shows 16.7%, of which 16.4 points lie between tasks that are presented to the model as independent. The other synthetic corpus comes in cleaner than reality at 0.3%, which is what rules out "synthetic corpora are like this" as the explanation and leaves it a property of that one release.
Before any of that, the raw files carry a more basic problem.
Two thirds of one corpus's rows and six sevenths of the other's are exact repeats of a name and description already in the file, against 0.4% for real tools. The consequence is stated plainly: any statistic computed over these releases without global deduplication measures how often the benchmark repeats a task, not how many tools it contains. If you have quoted a tool-count or a coverage number off one of these files, it was mostly counting repetition.
Take the paper's measured rates and point them at a number of servers you were thinking of enabling. Pick where you would source them, set how many, and read off how many boot, how many you would have to chase, and how many of the tools reaching your agent arrive with no machine-readable hint about whether they write.
The shape worth taking away is that the two frames differ by about eighteen points on both axes, in opposite directions — so a plan built on published ecosystem statistics will overestimate how many of your servers boot and underestimate how many of your tools arrive unlabelled. Budget a health check at enable time rather than at first use, and treat the annotation check as a per-server gate, because that is the level the decision was actually taken at.