Benchmark

A small, v0 pilot run against a public corpus — 25 mature open-source libraries with no known issue, 11 intentionally-vulnerable training apps, and 20 mature, deployed open-source web applications. Published with every caveat we found, not just the numbers that look good.

False-critical rate, libraries

0 of 25 clean repos, both tiers

Small sample: n=25, exact 95% CI 0–13.8%. Libraries only — applications are measured separately below. A point estimate, not a guarantee.

Verdict-level recall, Deep tier

100%

11 of 11 vulnerable apps verdicted unsafe

Verdict-level, not finding-level — parity with a union of three open-source scanners, not detection superiority. 91.7% under strict pre-registration.

Secret false-critical rate, applications

0 of 20 deployed apps, free deterministic tier

Down from a raw 35% via general precision rules, never per-repo allowlists — but the rules were tuned on these same 20 apps, so this is in-sample. Held-out validation is an open v1 item.

Application clean-pass, Deep tier

80%

16 of 20 deployed apps passed green

Model false-critical rate 0% — no critical on model opinion alone. All 4 not-green verdicts were adjudicated over-escalated on dependency drift; root causes since fixed, re-run outstanding.

Free vs. Deep Verification

NittiM ships two tiers. The free scan runs only deterministic checks — a secret scanner and a dependency-CVE lookup — no model in the verdict path. Deep Verification adds Claude Opus reasoning across the full audit framework, including the semantic vulnerabilities (SQL injection, authorization bypass, SSRF, logic flaws) that have no committed secret or vulnerable dependency to point at.

Tier	False-critical rate	Recall
Free — deterministic scan	0% (0 of 25)	63.6% (7 of 11)
Deep Verification — Opus	0% (0 of 25)	100% (11 of 11)

Free-tier mean hard-evidence findings per vulnerable app: 4.27. The free tier missed 4 of 11 apps — digininja/DVWA, juice-shop/juice-shop, stamparm/DSVW, OWASP/Vulnerable-Web-Application — because none of them had a committed production secret or a runtime high/critical CVE for the scanners to find. This is the precise gap Deep Verification exists to close, not a defect in the free tier.

Score separation, Deep tier

On the Deep tier, the readiness/safety score (0–100, higher is safer) separates the two populations clearly: a mean of 89.3 on clean libraries versus 30.3 on intentionally-vulnerable apps. The two populations are maximally contrasted by construction — audited libraries against deliberately broken training apps — so this gap describes the corpus, not how well NittiM discriminates a real app with two or three embedded issues from clean code, which this benchmark does not yet measure.

Applications — 20 mature, deployed web apps

The library numbers above answer the wrong question for most readers: NittiM audits applications, not libraries. This axis measures both tiers against 20 mature, widely-deployed open-source web apps (Ghost, Strapi, n8n, cal.com, Gitea, Mattermost, Discourse, Mastodon, Metabase, and others) across Node, Go, Python, Ruby, PHP, Elixir, and Clojure stacks. A deployed app ships a real dependency tree, so unlike a library, a not-green verdict here is not automatically a false alarm— every one was independently adjudicated against the OSV database and the repo's real lockfile before being counted.

Metric	Result
Secret false-critical rate (free tier)	0% (0 of 20)
Real runtime-CVE catches (true positives, reported separately)	3 of 20
Clean-pass rate (Deep tier, Opus)	80% (16 of 20)
Criticals on model opinion alone (Deep tier)	0% — zero across all 20

The unflattering number, published on purpose: of the five dependency findings that drove the 4 not-green verdicts, 2 were sound, 2 overstated, and 1 flatly false (a critical raised against an already-patched version because the scanner read the declared range instead of the installed lockfile version). The model never invented a critical — the failure mode was a deterministic layer escalating ordinary dependency drift too aggressively. Both root causes are fixed; the table reports the run as measured, before those fixes.

Application-axis caveats

Same rule as everywhere on this page: the qualifier ships with the number.

Mature apps are a lower bound, not the target class

The 20 apps are well-maintained, widely-deployed open-source projects — not the younger, thinner-reviewed, AI-assisted code NittiM is built to audit. A convenience sample of the best-run codebases on GitHub understates real-world error in both directions. Read every number on this axis as a lower bound.

One flatly false critical — found, disclosed, fixed at the source

Of five dependency findings driving the four not-green verdicts, two were sound, two overstated, and one flatly false: a critical raised against a package version the lockfile had already patched, because the scanner read the declared version range instead of the installed version. That scanner bug is fixed (installed versions are now resolved from pnpm/yarn lockfiles), so the false critical no longer fires — but the table above reports the run as measured, not as it would look after the fix.

Over-escalation was the real failure mode — since retuned

All four not-green verdicts penalized ordinary dependency drift rather than a reachable defect in the app's own code. The pure model never invented a critical; the deterministic dependency floor was too aggressive. It has since been split by severity — a critical runtime vulnerability still forces the harshest verdict, a high-severity-only one no longer does (it can never be waved green either) — and dev/test-only dependencies no longer count toward production risk. A post-fix re-run on held-out apps is an outstanding v1 commitment.

Single pass per app — verdict variance unmeasured

Each app was audited once. One model-only not-green (Metabase) flipped to green on an independent re-audit, so run-to-run variance is real and these figures understate it. The v1 commitment is to report verdict variance across N≥3 runs; until then, treat any single verdict near a band boundary as potentially unstable.

The adjudicator shares a model family with the system under test

Every adjudication was performed by an Opus-family model — the same family NittiM ships — because no strictly stronger independent judge exists. Verdicts were decided against OSV and real lockfile ground truth rather than model opinion, which blunts but does not eliminate shared-family blind spots. Discount the soundness numbers accordingly.

Read this before the numbers above

This is a v0 pilot on a small, hand-built corpus. Every limitation we found is listed here, next to the numbers it qualifies — not in a footnote.

Small sample — point estimate, not a tight bound

n=25 clean repos, n=11 vulnerable apps. At n=25 with zero observed false criticals, the exact 95% confidence interval for the true false-critical rate is 0%–13.8% — a single false critical would move the observed rate to 4%. This is a v0 pilot, sized to catch gross failure modes, not to certify a precise rate.

Library FCR and application FCR are different measurements

The 0% false-critical rate above is measured on mature open-source libraries, frameworks, and CLI tools. Web applications — the code NittiM is built to audit — are measured separately on a 20-app corpus (see the application axis below), where a deployed app's real dependency tree makes "false alarm" a genuinely harder call. Neither corpus covers the young, AI-assisted codebases NittiM actually targets; both are lower bounds.

Recall is verdict-level, not finding-level

A 100% recall figure means every vulnerable app in the corpus was verdicted unsafe — not that every documented vulnerability in each app was individually located. On this corpus, verdict-level recall is parity with a one-line rule unioning the outputs of three open-source scanners (Gitleaks, Trivy, Semgrep), not evidence of detection superiority over them.

The juice-shop-ctf disclosure

A 12th vulnerable-corpus entry was the only miss in the initial run, then removed afterward on the grounds that it is a CTF-challenge generator, not a vulnerable application. The removal is defensible but happened after the result was known, so we publish both numbers: 100% (11 of 11) on the corpus as currently defined, and 91.7% (11 of 12) under strict pre-registration with that entry kept in.

The juice-shop-ctf entry, both ways

Re-frozen 11-app corpus: 100% (11 of 11). Strict pre-registration with the disputed 12th entry kept in: 91.7% (11 of 12).

Confidence interval, not a point claim

The exact 95% confidence interval on the clean-corpus false-critical rate is 0%–13.8% at n=25. Treat 0% as a small-sample point estimate, not a guarantee.

The full corpus, selection predicates, protocol, and methodology are public: github.com/ilanwolberger/fis-benchmark. See Data & Trust for what NittiM stores, or go back to the home page.