Someone else's corpus,someone else's scorer.

Disclosureread this before quoting the number

This number is not blind, and it has not been for three rounds.

The first two runs were blind: the engine had never seen this corpus, and 12.5 is what it scored on code it had not read. Every run after that was measured once the benchmark's own false negatives had been read and the missing rules written.

Every rule added is of a kind any SAST ships — weak PRNG for tokens, cookie flags, CSRF exemptions, a committed fallback signing key, open redirect, credentials in logs — so no rule is fitted to a particular fixture. The selection, however, was corpus-informed three times, and the advantage compounds. The gap between 12.5 and 61.2 is the size of it. 12.5 is the conservative figure for an unseen codebase, and the honest successor to this measurement is a benchmark this repository has not seen.

Run history30 runs, one corpus

The cost of each round, alongside its score.

Same corpus, same scorer, same clone. Each round re-scores the previous engine on the current checkout first and reproduces its committed figures digit for digit, which is what makes the movement attributable to the engine rather than to the corpus.

Precision rises alongside recall in every round after the first. The repository treats that as evidence of rules rather than curve fitting; a round in which precision falls is treated as a signal to narrow the rule rather than to keep it.

Baselines

Against what the benchmark itself publishes.

RealVuln publishes a best-reported figure per category. Three of these four rows are theirs; the bold one is this engine, measured by their scorer on their corpus. Every figure on this page keeps the scorer's own units — F3 on 0–100, precision and recall as fractions — so the rows stay comparable to the published table. The landing page states the same measurements as percentages throughout.

Above rule-based SAST on both metrics, and well below both a general-purpose model and the purpose-built system — both of which were measured the way this one no longer is. The supportable claim is “above Semgrep on a corpus since read”, not “better than Semgrep”.

Where the recall goes

Every family, largest labelled pool first.

The ordering carries more information than the bars: the four families at the top hold more than half of every label in the corpus, and they are the four this tier scores worst on. A score is determined by the large pools rather than by the ones a scanner finds easy.

• other536 / 83165%
• sensitive_data_exposure61 / 14143%
• security_misconfiguration90 / 10883%
• xss49 / 9850%
• broken_access_control2 / 763%
• missing_auth7 / 749%
• sql_injection67 / 7194%
• hardcoded_credentials25 / 5248%
• command_injection39 / 4685%
• denial_of_service31 / 4470%
• open_redirect37 / 4092%
• path_traversal23 / 3959%
• ssrf16 / 3743%
• xxe32 / 3689%
• insecure_deserialization27 / 3479%
• code_injection28 / 3093%
• xpath_injection0 / 40%
• http_header_injection0 / 10%

The pattern classes are near their ceiling: command injection, open redirect, code injection and XXE all exceed 80%. The pools that determine the score are not pattern classes. Roughly 130 of the labels in sensitive_data_exposure and security_misconfiguration sit on a function definition, the same shape as broken_access_control and missing_auth. Those require the business-logic pass rather than a further rule, and no rule added here has reached them.

Per repository62 of 66

All of them, best first — including the ones that scored nothing.

Ranked by F3. Published in full rather than as a top five, because the top five on this project's own page was two rounds stale before anything compared it to the scorer output, and a leaderboard nobody reconciles flatters itself.

0 of 62 repositories scored 0.0 — nothing labelled was found in either. They are in the table, at the bottom.

The other languagetwo runs

What the JavaScript side does, and what it did before it had read this.

Every figure above describes Python. This one describes JavaScript, and it has been measured twice. The first run was blind — no rule in this engine had been written or chosen by reading a label from this benchmark — and it scored 0.2286. That is the figure for unseen code and it does not improve. What the blind run bought was a diagnosis; the number here is what the engine scores after acting on it. The two corpora are not comparable in either direction — that one is applications, this one is libraries, and a library has no request handler for the reachability analysis to start from.

• redos28 / 8732.2%
• prototype-pollution72 / 18538.9%
• command-injection62 / 10161.4%
• code-injection22 / 3366.7%
• path-traversal128 / 16776.6%

The blind run said three things and this is what each became. ReDoS scored zero because that analysis was Python-only — stated in the roadmap before a package was fetched, so it could not afterwards be presented as a discovery; it reads JavaScript now, and 8 of 87 is the honest size of criteria that describe exponential backtracking when most real reports are polynomial. Prototype pollution is still the worst class, but the rule that scored the nine has been retired for one that reports the write rather than the loop. And 35 sinks under dist/ or build/ were called a scoping decision rather than a detection failure — the engine reads those directories now when the package publishes them, 29 became reachable, and two were found. That claim was the flattering half and it was wrong.

What this does not say

Three things this number is not.

• 1

  It is not a measurement of the LLM tier

  Tier 1 is the tier meant to reach the classes the deterministic one structurally cannot. Measuring it needs paid inference across every repository here, and it is a stated non-goal rather than backlog. The tier ships off by default and the claim was narrowed instead.

• 2

  It is not a measurement of the JavaScript engine

  RealVuln v1 is Python-only. The structural analyses grew a JavaScript/TypeScript front end, so the same four questions are now asked of both languages and only one of the two answers has ever been scored by someone else. The JS side ships as a regression floor, not as a score.

• 3

  Not a claim about any other repository

  The families holding most of the labels are the ones still near zero, and no pattern added here touches them. A corpus of deliberately vulnerable applications is not the shape of a production codebase either — in both directions.

None of these is a caveat added after the fact. Each one is a bound this measurement has had from the first run, and the report the tool writes states its own bounds the same way.