Every established way to audit software needs something you do not have for a hosted tool server: the source, a binary, or the ability to poke the running thing and watch what happens. This paper proposes auditing from metadata alone — for each tool, only its name, its natural-language description and its input schema — by asking which data flows every implementation consistent with that description would have to contain, and writing out the resulting attack scenario rather than an exploit. Across 20 widely deployed servers and 177 sampled tools, it flagged 143 and recovered 94 of the 95 a human team later confirmed exploitable, against 80 for a single-prompt baseline, at about $1.65 a server. The bill comes as precision: 65.7% against the baseline's 82.5%, and a quarter of its scenarios did not survive a look at the source.
White-box analysis reads the source or the binary. Black-box analysis calls the exposed interface and watches the responses. Grey-box analysis executes the target and follows coverage feedback. They differ in how much they see, but they share a prerequisite: an analyst who can observe or interact with the thing being analysed.
That prerequisite fails for exactly the case that matters here. A third-party tool server you are about to enable is typically closed-source, hosted by someone else, gated behind a paid plan, or simply not something you are entitled to probe — and probing a live production server is how you get a phone call. Step through the four settings and what each one needs.
The no-box setting keeps only what a server publishes in order to be usable at all. A tool has to describe itself to be callable, and that description is enough to constrain what it must do — which is the whole bet.
The mechanism is a single move. If the metadata says a tool fetches a page and returns its content to the caller, then every implementation consistent with that sentence contains a path from untrusted remote text into the model's context. The paper calls that an irreducible data flow: the minimal relationship no conforming implementation can avoid.
What the flow does not tell you is whether the vulnerability is real. The paper's own worked example is a plain filter that selects database records: the description forces a path from user-controlled input into a query, and that path is what makes injection relevant — but an implementation may build the query by string concatenation, by a parameterised statement, by a restricted parser, or through an object mapper, and only the first is exploitable. Step the implementations.
So the output is deliberately not a proof of concept. It is what the authors call a theory of concept: a named attack scenario stating who the attacker is, what they control, where they would plant the payload, how faithfully the tool would carry it, and what the impact is — together with the implementation assumptions it rests on. It is the strongest claim an analyst with no access can honestly make, and it is a claim about a class of implementations rather than about the one you are running.
All four have to hold at once. A flow that carries attacker text verbatim into a context window is only a finding if the attacker can reach the source, if what arrives reads as instructions rather than as an identifier, and if nothing on the path neutralises it. That last one is rarely a rescue: of the tools examined, 92.3% apply no sanitisation at all on the path from external data to the model.
The interesting cases are the ones where the hypothesis and the evidence come apart. Both of these were produced from registration metadata alone, before anyone looked at an implementation. Step through them, and switch between what metadata alone established and what the authors found when they went looking.
The second case is the honest one to sit with. A group-management tool that updates a name or a handle also returns the group's existing description, which somebody else may have written — so instructions planted in a description reach the model when an unrelated user edits the group. Reading the source supported the flow. Building an actual exploit needed a write-scoped token on a paid workspace, so it was never run, and the scenario is where the analysis stops. That is not a gap in the method; it is the method's ceiling, and the point is that the ceiling is still above nothing.
The comparison is against the obvious thing you would do instead: hand the same metadata to the same model in one prompt, with the same threat model and the same rubric, and ask for a verdict. The pipeline's extra machinery — infer the entities, assemble the flows deterministically, then judge each flow — is what the extra recall is being attributed to. Pick a metric.
Recall is the only figure this experiment pins down exactly, because the 82 flagged tools that were never confirmed are not established negatives — a tool goes unconfirmed either because it has no injection channel or because no safe trial could be run against it. Both precision figures are therefore lower bounds. But the paper does supply its own upper-bound signals, by going back over all 143 flagged tools with progressively better evidence.
Read together: with the source in hand, a quarter of the scenarios were judged implausible, and when the flows were exercised with benign inputs, 11.2% did not reach the context at all — those cluster in write-only and interaction tools that return nothing but a confirmation. So somewhere around a quarter to a third of what the metadata-only pass flags is wrong, not merely untested. That is the real shape of the tool, and it is still a useful shape.
The use this supports is triage, not judgement. Set how many third-party servers you are looking at, choose how carefully you would screen, and read off how many tools land in the queue and roughly how much of that queue is worth a human's afternoon.
The trade to notice is that the two approaches are not ordered. The cheap single prompt hands you a shorter, cleaner list; the full pipeline hands you a longer, noisier list that misses almost nothing. If the queue feeds a human reviewer, the longer list is the right one, because a missed channel costs more than a wasted review. If it feeds an automated block, it is the wrong one.