The Benchmark That's Half Traps — and Why That's Brilliant

There's a test suite for security scanners where 47% of the test cases are
deliberately designed to make your tool look stupid.

Not edge cases. Not ambiguous code. Purpose-built decoys — code that displays
every symptom of a vulnerability while being completely safe. If your scanner
flags them, you lose points.

When I first read that, it sounded almost hostile. Then I ran my own scanner
against them, watched it walk into trap after trap, and understood: that 47% is
the most valuable part of the entire dataset.

This article dissects two real traps — including one my scanner falls for right
now — and explains why a benchmark that punishes you this way is the best thing
that can happen to a security tool.

What the OWASP Benchmark is

The OWASP Benchmark is a generated Java application containing thousands of small
servlets, each one a self-contained test case. Every case is labelled: either it
contains a real, exploitable vulnerability, or it doesn't.

That labelling is what makes it valuable. Most security testing is
unfalsifiable — you scan a real codebase, get 40 findings, and have no idea how
many real bugs you missed. Here you know the answer for every case, so you can
compute real numbers.

In the four vulnerability classes my scanner handles, the counts are:

1,478 cases. 701 traps. Path traversal has more decoys than real bugs.

Why does anyone need a labelled test set?
Two scanners disagree constantly. Without ground truth you can't tell whether
tool A found something tool B missed, or whether tool A is just noisier. With
labels, you can compute precision (of what I flagged, how much was real?) and
recall (of the real bugs, how many did I catch?) — and a tool that inflates one
at the expense of the other gets caught immediately.

How these cases are made

The Benchmark isn't hand-written. It's generated — a template engine combines a
source, an optional transformation, an optional sanitiser, and a sink, then stamps
out a servlet for each combination. That's why the files look so mechanical, and
why there are thousands of them.

This generation is exactly why the decoys are convincing. They aren't "safe code
that happens to look odd." They are the same template as the vulnerable case
with one component swapped: a real sanitiser instead of no sanitiser, or a
constant-returning source instead of a real one. Structurally the vulnerable and
safe versions can be nearly identical — which is precisely the discrimination
problem a scanner has to solve.

It also means the traps come in families. Learn to recognise a family and you
understand a whole class of scanner failures at once. Here are the two most
instructive.

Trap type 1: the input that was never input

Here's a real test case, BenchmarkTest00052. Read it as a scanner would:

org.owasp.benchmark.helpers.SeparateClassRequest scr =
        new org.owasp.benchmark.helpers.SeparateClassRequest(request);
String param = scr.getTheValue("BenchmarkTest00052");

String sql = "{call " + param + "}";

java.sql.CallableStatement statement = connection.prepareCall(sql, ...);
java.sql.ResultSet rs = statement.executeQuery();

Everything about this screams SQL injection. A wrapper object built from the HTTP
request. A getter called on it. The result concatenated directly into a SQL
string. That string executed.

Every structural signal says vulnerable.

Now here is getTheValue, in a different file:

public String getTheValue(String p) {
    return "bar";
}

It returns a constant. It ignores its argument entirely. The value reaching that
SQL query is the literal string "bar", every single time, no matter what the
attacker sends.

There is no vulnerability. There was never any user input in the flow at all.

My scanner falls for it

I checked. Here is my own tool's output on this exact case:

My scanner flags BenchmarkTest00052 (a labelled DECOY): 2 time(s)
  class: sql-injection
    line  46  scr.getTheValue("BenchmarkTest00052")
    line  46  param
    line  48  "{call " + param + "}"
    line  48  sql

Twice, in fact — once as SQL injection and once as XSS.

Why? Because my source list includes SeparateClassRequest.getTheValue. I added
it deliberately: the Benchmark uses that wrapper in hundreds of genuinely
vulnerable cases, and without it my recall collapses. So I treat it as a source of
untrusted data.

In this one case, that assumption is wrong. The method returns a constant, and I
report a vulnerability in code that cannot be exploited.

This is not a bug I can fix by being more careful. It's the fundamental
trade-off of the technique: I either treat that method as dangerous (catching
hundreds of real bugs, plus this false alarm) or I don't (avoiding this false
alarm, missing hundreds of real bugs). Taint analysis works on structure, and
structurally these cases are indistinguishable.

Trap type 2: the filter that actually works

The second flavour is more subtle. BenchmarkTest00282:

String param = "";
java.util.Enumeration<String> headers = request.getHeaders("Referer");
if (headers != null && headers.hasMoreElements()) {
    param = headers.nextElement();               // genuinely attacker-controlled
}
param = java.net.URLDecoder.decode(param, "UTF-8");

String bar = org.owasp.esapi.ESAPI.encoder().encodeForHTML(param);

response.getWriter().printf("Formatted like: %1$s and %2$s.", obj);

This one is different in an important way: the input really is
attacker-controlled. It comes from an HTTP header the client controls
completely. It really does travel to a sink that writes into the HTTP response.
A path from source to sink genuinely exists.

The line that saves it is ESAPI.encoder().encodeForHTML(param). ESAPI is the
OWASP security library, and encodeForHTML converts <script> into
<script> — the browser renders it as text instead of executing it. After
that call the value cannot form an HTML tag.

For a path-tracing tool this is genuinely hard. The data flows. The path is
complete. The only thing standing between this and a real XSS bug is what one
function does to the value on the way through — which is a question about
meaning, not structure.

The three ways a tool can handle this

Option A: model every sanitiser by hand. Maintain a list — ESAPI's encoders,
OWASP Java Encoder, Spring's utilities, Apache Commons, and whatever your
codebase wrote itself. When a flow passes through one, break it.

This works, and it's what mature tools do. It's also a permanent maintenance
burden that is never complete. Every library, every version, every home-grown
sanitize() helper needs an entry. Miss one and you get false alarms; get one
wrong and you hide a real vulnerability, which is far worse.

Option B: report everything and let humans sort it out. Cheap, safe in the
"never miss a bug" sense, and the reason security tools are famous for crying
wolf. Developers who wade through 40 alarms to find 3 real ones eventually stop
reading the alarms.

Option C — what I'm testing: deliberately over-report in the mechanical stage,
then hand each finding to something that can read code and ask "does this filter
actually filter?"

That's my whole project. And this benchmark is the ideal test of it, because 701
of its cases are exactly that question, asked 701 different ways.

How my scanner actually does against the traps

Talking about traps in the abstract is easy. Here is my scanner's real
false-positive rate per category — the fraction of decoys it wrongly flags:

My deterministic layer falls for 88% of the traps. That is not a
typo, and it is not a disaster — it's the designed behaviour of a layer built to
over-report. But it does show, concretely, how little a pure taint-reachability
analysis can do about this problem. The traps work.

For comparison, on the same decoys:

Both mature tools do better than mine, and neither is close to clean. CodeQL —
which finds every single real bug — still flags 427 safe cases. Even the best
freely available taint engine falls into 6 out of every 10 traps.

That's the sentence I'd underline for anyone who thinks false positives are a
solved problem, or a sign of a badly built tool. They're a property of the
technique.

Going deeper (skip if you just want the lesson)

There's a third trap style worth knowing about, because it defeated my scanner
in a way I didn't anticipate. Some cases route data through a helper method
that assigns a constant under a condition that is always true:

int num = 106;
bar = (7 * 18) + num > 200 ? "This_should_always_happen" : param;

7 × 18 + 106 = 232, which is always greater than 200, so bar is always the
constant. The tainted param never survives. A human reads this in ten seconds;
a taint tracker doesn't do arithmetic, so it follows the false branch and
reports a flow.

What makes this one instructive is that my AI judge also got it wrong at
first — not because it couldn't reason about the arithmetic, but because my code
slicer cut the snippet at line 100 and the assignment was at line 103. The judge
never saw the decisive line. That's a whole article of its own, later in this
series.

Why the trap ratio is the right design

A test suite of only real vulnerabilities would be trivially gamed. Flag every
line that touches user input and you'd score 100% recall. Useless, but perfect on
paper.

The 701 decoys make that strategy fail loudly. They convert the benchmark from
"can you find bugs?" — easy — into "can you tell the difference?" — which is
the actual job.

They also mirror reality. Real codebases are full of input that reaches sinks
after passing through validation, because developers do write validation. A tool
that can't recognise a working filter isn't slightly noisy in production; it's
mostly noise.

And the ratio has a pointed implication for anyone building AI security tools: if
your evaluation set is all real bugs, you have not tested the thing that matters
most. An LLM that confirms every finding it sees will score beautifully on a set
with no decoys and be worthless in front of a developer.

How to steal this design for your own evaluation

The Benchmark's structure transfers to almost anything you need to evaluate — a
classifier, an LLM judge, a moderation filter, a fraud check. Three properties are
worth copying:

1. Near-miss negatives, not obvious ones. The decoys aren't random safe code;
they're the vulnerable template with one component changed. If your negative cases
are wildly different from your positives, you're measuring whether the system can
tell apart things that were never confusable. Build negatives that differ from
positives by one meaningful thing.

2. Enough negatives to matter. At 47%, a "flag everything" strategy scores
terribly. At 5%, it would still look respectable. The ratio determines whether
your metric can even detect the failure mode you care about.

3. Families, not one-offs. Grouping decoys into recognisable types (constant
source, working sanitiser, dead branch) means a failure tells you which
capability is missing, not just that something went wrong. My 88% trap-failure
rate isn't actionable on its own; "my taint analysis cannot evaluate conditions or
recognise sanitisers" is.

The mistake I see most often in AI evaluation is a test set of things the system
should say yes to, plus a handful of obvious no's. That set cannot distinguish a
good judge from a rubber stamp — and as I found when I compared three AI judges on
identical inputs, the spread between them is enormous.

What I learned

1. A benchmark that only rewards finding things measures the easy half. The
hard half is discrimination. Whatever you're evaluating — a scanner, a classifier,
an LLM judge — count how many of your test cases are supposed to come back
negative. If the answer is "few", your numbers are inflated.

2. Some false positives are structural, not sloppy. My scanner's failure on
getTheValue isn't carelessness. It's the cost of a source list that catches
hundreds of real bugs. Understanding which of your errors are trade-offs and which
are defects tells you where effort actually pays.

3. Test data that makes your tool look bad is worth more than test data that
flatters it. I could have evaluated on my own hand-written samples and reported
excellent numbers. The 701 traps are where I learned what my tool actually does.

Next in this series: the results. My scanner, Semgrep, and CodeQL — same 1,478
cases, same scoring program, all three numbers published, including the one where
I lose.

I'm Ali Afana — AI builder and security researcher, writing from Gaza. I
build systems in public, measure them against ground truth, and keep the
receipts. This scanner is one project on a longer road — follow for what
comes next.

GitHub · X · LinkedIn