Every application security vendor claims to cut noise. Very few say by how much, on what, with which settings, or how you could check. We think a claim you cannot reproduce is a marketing line, not evidence. So we ran InsightFortify on six public, deliberately vulnerable applications with nothing tuned, published the per-repository numbers, and wrote down exactly what the benchmark does and does not measure.
In short
- 1,216 raw tool results became 844 unique findings and 284 actionable ones: 77% fewer items to act on, across six public applications, with default settings.
- The reduction comes from two places: correlation (the same issue from several tools, files or commits becomes one finding) and context (dev-only, transitive, test-code and exploitability factors move items out of P1/P2).
- Default context understates the result. Nothing was marked internet-facing, sensitive or compensated, so business context, the biggest lever in a real estate, played no part.
- It is reproducible. The script, the repository list and the raw results are in the product repository; the same command produces the same table.
- A load test sits beside it: 200,000 findings, 20 concurrent console users, 75 requests a second, p95 0.70 seconds, zero errors, on a laptop.
Why publish a benchmark at all
Buyers of application security posture management are asked to believe a number they cannot verify until after the contract is signed. “Up to 95% noise reduction” appears in most category marketing, usually without the denominator. Was it measured on a real estate or a demo? Were the duplicates removed before the count, or after? Was the business context tuned by a professional services team for six weeks first?
A CISO who has to defend a tooling decision to a CFO or an audit committee needs something firmer: a number with a method, on inputs anyone can obtain, that can be reproduced by someone who does not work for the vendor. That is what this post provides.
One headline figure, no method, no inputs, no way to check. Usually measured on a hand-picked customer after tuning, sometimes on synthetic data.
Public inputs, default settings, the scanner versions stated, raw per-tool counts published, and the command that reproduces it.
The method
The benchmark script runs the product’s own worker pipeline — the same scanners and the same ingest that a customer deployment runs — against each repository in a throwaway organization. It records three numbers per repository:
- Tool raw: what each scanner printed. Trivy vulnerabilities, misconfigurations and secrets; Gitleaks leaks across the full git history; Checkov failed checks; the AI and API analysers’ results.
- Unique: after correlation. The same CVE in the same package reported by two tools, the same secret found in several commits, the same misconfiguration in a file scanned twice, each becomes one finding with its sources listed.
- Actionable: open findings at the organization’s actionable priorities, which by default means P1 and P2. This is the list a security team would ask developers to work on this sprint.
full history"] --> B["Worker scanners
Trivy, Gitleaks, Checkov,
AI and API analysers"] B --> C["Tool raw
what each tool printed"] C --> D["Correlate
fingerprint, merge sources"] D --> E["Unique findings"] E --> F["Score with
default context"] F --> G["Actionable
open P1 + P2"] style G fill:#ccfbf1,stroke:#0d9488
The benchmark runs the same pipeline a deployment runs. Nothing is special-cased for the test.
Settings: deliberately untuned
Every repository was scanned with default context. None was marked internet-facing, none was given a data classification or a business criticality, no compensating control was recorded, and no finding was dismissed or snoozed. This matters, and we come back to it below: in a real deployment, those settings are where most of the reduction comes from.
The results
| Repository | Tool raw | Unique | Actionable | Reduction |
|---|---|---|---|---|
| OWASP NodeGoat | 340 | 215 | 37 | 89% |
| OWASP Juice Shop | 446 | 339 | 95 | 79% |
| DVWA | 29 | 29 | 11 | 62% |
| OWASP WebGoat | 138 | 109 | 73 | 47% |
| OWASP RailsGoat | 234 | 123 | 57 | 76% |
| django.nV | 29 | 29 | 11 | 62% |
| Total | 1,216 | 844 | 284 | 77% |
Duplicates are merged, not discarded; deprioritised findings stay in the queue with their reasons.
Reading the table
The spread is the interesting part. WebGoat drops only 47%, because it is a teaching application full of directly used, deliberately vulnerable dependencies and secrets: there is less to correlate and little test code to discount. NodeGoat drops 89%, largely because its dependency tree is deep and old, so many findings are transitive or development-only and the score moves them down. DVWA and django.nV are small, and every unique finding is a distinct issue, so correlation removes nothing and the reduction comes from scoring alone.
None of these numbers means a finding was deleted. A merged duplicate becomes a source on the surviving finding. A deprioritised finding stays in the queue at P3 or P4 with its factors listed, so an auditor or an engineer can see why it is not at the top.
What “actionable” means here
Open findings at the organization’s actionable priorities: P1 and P2 by default. It is a setting. A team that only acts on P1 would see a far larger reduction; a team that acts on P1 to P3 a smaller one. We report the default because it is what a new deployment shows.
Why the real number is usually larger
The benchmark ran with no business context, which is the opposite of how the product is meant to be used. In a real estate, a security lead marks which services face the internet, which handle payment or personal data, and which sit behind a web application firewall. Those factors move findings in both directions: an internet-facing payment service rises; an internal batch job with no sensitive data falls. For most estates, the majority of repositories are internal, so context tends to remove more items from the actionable list than it adds.
Our own demo estate, with a little context set, shows the effect on a smaller sample. We prefer to lead with the untuned public number, because it is the one a sceptic can reproduce.
What the benchmark does not measure
- Static code analysis. The machine that ran it had no SAST rule pack configured, so Opengrep was skipped. Code findings would add both raw results and actionable ones; we expect the correlation gain to be smaller there, because code findings are rarely duplicated across tools.
- Reachability gains on deep dependency trees. Function-level reachability only lowers a score with complete evidence, and a transitive dependency is classified as unknown. On these applications, where most vulnerable packages are transitive, reachability contributes little. It does more on services with flatter dependency trees.
- Accuracy of each finding. The benchmark counts what the scanners report; it does not grade whether each report is a true positive. That is the scanners’ job, and why we use open-source engines you can run yourself.
- Your estate. Six teaching applications are not a bank’s monorepo. The right test is to run the same script on two of your own repositories.
How to reproduce it
PostgreSQL, Redis, the API and a worker, with the pinned scanner images.
The script refuses a database whose name does not say test, bench or load, so it cannot touch real data.
Pass repository keys (nodegoat, juice-shop, dvwa…) or none for all six; it clones, scans and ingests each one.
Per-tool raw counts, unique, actionable, the priority split and the scanner versions, written to the output folder.
Add two internal repositories and set their context the way you would in production.
And at scale: the load test
Noise reduction is only useful if the console stays fast when the queue is large. The load-test harness generates a synthetic estate and drives the console’s real request mix: queue pages, filters, search, facets, finding detail, the dashboard and the cloud pages.
| Run | Data | Requests / s | Overall p95 | Dashboard p50 | Errors |
|---|---|---|---|---|---|
| Before (Python-side counting) | 200,000 findings, 20 clients | 40 | 1.57 s | 1.47 s | 0 |
| Shipped (SQL counting + indexes) | 200,000 findings, 20 clients | 75 | 0.70 s | 0.65 s | 0 |
The first run found two pages, the dashboard and compliance, doing aggregation in Python over tens of thousands of rows. Moving the counting into PostgreSQL and adding five indexes for the queue’s hot paths nearly doubled throughput. The test ran on a laptop that was building other software at the same time, so the absolute times are pessimistic; the comparison between runs is the point.
Not yet measured
Write-heavy load (large scans ingesting while people browse), multi-million-row organizations, and PostgreSQL on separate hardware. We will publish those numbers when we have run them, not before.
What correlation actually merges
“Deduplication” is often described as if it were trivial. It is not, and the details decide whether a team trusts the result. Three kinds of merge account for nearly all of the 372 results removed in the benchmark:
| Merge | Example | Why it is safe |
|---|---|---|
| Same advisory, same package, several tools | A CVE in a JavaScript package reported by Trivy from the lockfile and again from the SBOM | The CVE id (then the GHSA id) and the package URL are the identity; the version and location are kept on the finding |
| Same secret, several commits | One API key committed, moved and deleted across a dozen commits in the history | The secret is fingerprinted by a hash of its value, never stored in clear, so each commit becomes a source, not a finding |
| Same misconfiguration, re-scanned | A Terraform bucket without encryption found by two scans of the same branch | Rule, file and resource form the fingerprint; a rescan updates the finding instead of adding one |
Two rules keep merging honest. A finding reported by several tools is only marked fixed when the last of them stops reporting it, so one tool going quiet cannot close a real issue. And when a later tool lacks a field another supplied — an installed version, a fixed version — the merge keeps what was already known rather than blanking it.
How to run a fair vendor bake-off
If you are comparing application security posture platforms, the same method works for every vendor on the shortlist. The point is to hold the inputs constant and make each vendor show its working.
- Pick the inputs. Two or three of your own repositories, one with a deep dependency tree and one with infrastructure code, plus one public application from the table above so results can be compared with ours.
- Fix the settings. Run once with defaults, then once with the same business context entered in every product: which services face the internet, which hold sensitive data.
- Ask for three numbers per repository: raw tool results, unique findings, and the list a developer would be asked to work this sprint.
- Sample the bottom of the list. Pick ten findings each product deprioritised and ask why. A product that cannot explain its own ranking will not be able to explain it to a developer either.
- Time the console. Load the queue with a realistic number of findings and measure page loads, not just scan throughput.
A question worth asking every vendor
“Show me a finding your product moved down, and the exact reason, on my code.” Explainable deprioritisation is what lets a security team defend a shorter list to an auditor.
What it means for each reader
For the CISO
A vendor number you can defend: public inputs, a stated method, a reproducible command.
Ask every vendor on your shortlist for the same thing, on the same six repositories.
For the AppSec lead
Expect correlation to remove a quarter to a third of raw results, and context to do the rest.
Set exposure and data classification first; it is the biggest lever and the cheapest to maintain.
For the board
A developer team asked to work 284 items instead of 1,216 can actually finish its list.
Finishing is what moves the SLA and the risk number, not scanning more.
A noise-reduction claim without its inputs is not evidence.
Ours is 1,216 → 844 → 284 on six public applications, untuned, and you can run it yourself.