A "lexical query" is not the same thing as a query that needs lexical matching.

My retrieval benchmark sorts every query into two groups: ones that contain an identifier — a file path, a version number, an error code — and ones that don't. Queries with identifiers are supposed to be the easy case for keyword search. They were the case where it lost by the widest margin.

The reason: for 54% of those queries, the identifier never appears in any of the documents that answer them. There is nothing for exact matching to match against.

The usual advice says keyword search handles keyword queries, dense retrieval handles the rest, and hybrid gives you both. That advice is why most people run hybrid search. None of it held up here, and the part I was confident about is the part that broke worst.

The setup

BEIR / CQADupStack, the unix and android subforums: 70,380 documents and 1,771 judged queries. The task is finding duplicate questions — given a Stack Exchange question, find the earlier question that asks the same thing.

Before any retrieval runs, a classifier labels each query lexical or semantic. It only looks for lexical signals: something shaped like an identifier, or text in backticks. Semantic is just everything left over. There's no equally reliable way to look at a query and tell that it needs meaning-based matching, the way you can tell that a string looks like a hex literal. 214 queries end up labeled lexical, 1,557 semantic.

The BM25 side uses Porter stemming and identifier-aware tokenization. Anserini's Lucene analyzer, which produced the published BEIR numbers I compare against, stems by default, and the dense side gets that kind of robustness for free from how it tokenizes. The unstemmed numbers are in the repo as a control.

The result

Recall@3, macro-averaged:

                              overall   lexical   semantic
BM25 (identifier-aware
      + Porter)                0.2255    0.3070     0.2143
Dense (all-MiniLM-L6-v2)       0.3834    0.4842     0.3695

Dense wins overall, which is no surprise — that roughly matches the published BEIR ordering, and the dense nDCG@10 of 0.4071 is close enough to the reference number for this model that I trust the harness.

The surprise is where it wins. Dense beats BM25 by +0.18 on the lexical layer and +0.16 on the semantic one. The gap is bigger on exactly the queries full of file paths and version strings. A paired bootstrap over 10,000 resamples keeps the lexical interval clear of zero, and a sign test on the overall comparison gives 429 wins against 74 losses, p = 8.1e-62. It also holds when I run the two subforums as separate datasets instead of one merged pool (unix +0.2202, p = 3.4e-07; android +0.1940, p = 2.2e-05), so it isn't an artifact of merging them.

Why

That 54% explains the whole thing. BM25 can only match xorg.conf against documents that literally contain xorg.conf, and in a corpus of duplicate questions, most documents that do are about somebody else's unrelated problem. The question that actually duplicates yours describes the same situation but never types the path — or writes /etc/X11/xorg.conf, or
a different version number, or leaves it out. Worse, a rare token gets a very high IDF, so on the rare occasion it does match, it dominates the score and pushes up a document whose only connection to the query is that one string. The bi-encoder ignores the identifier entirely and matches on the problem description around it, which the duplicate really does
share.

So the classifier does exactly what it was written to do, and still measures the wrong thing. It tells you whether a query contains something identifier-shaped. I had been reading it as "this query needs exact matching." Those turn out to be different questions,
and on this task the answers differ more than half the time.

How far this goes: finding duplicate questions is close to the worst case for the assumption that identifiers are shared, because the query and the target are two different people describing one problem in their own words. In log search or code search the identifier is shared by construction, and I'd expect the usual advice to hold there. So the finding isn't "dense beats BM25 on lexical queries." It's that a query containing keywords and a query helped by keyword matching are two different things — and if you're routing between retrievers based on what a query looks like, you're assuming they're the same thing.

The useful version: how often a query's identifiers show up in its own correct answers is something you can measure on your corpus before building anything, and it tells you whether a keyword arm is worth having.

The check that made this believable

A benchmark that calls every difference real is worthless. So I pointed the same statistics at a comparison where I expected to find nothing.

That comparison was my identifier-aware tokenizer. It emits everything the plain one does, plus the full identifier — strictly more information, so it should only ever help, but in fact it didn't. On the lexical layer it landed 0.0125 below the plain tokenizer, with a confidence interval spanning zero and only 10 of 214 queries changing at all. p = 0.75. No
effect, and the same non-result shows up in both subforums separately.

That's what makes the p = 8.1e-62 above worth anything. The tests can tell a real difference from an imagined one.

Three limits on all of this:

• The bootstrap resamples queries. It covers variation in which questions I happened to test, not in the model, the preprocessing, or the relevance judgments.
• One embedding model, one corpus, one task. The 54% is a fact about the dataset that no choice of model would change, but everything built on top of it was measured on a single setup.
• These comparisons came after the fact. They show the ordering here isn't luck. They don't show it transfers anywhere else.

The repo has the rest of it: why equal-weight RRF ends up worse than dense alone, a tokenizer change that quietly cost recall, a leakage audit of CQADupStack, and the per-subforum replication. Code, full per-layer metrics, and the significance script

./scripts/download_data.sh     # ~65 MB into data/cqadupstack/
python -m pytest tests/ -q
python run_eval.py             # BM25 rows only, no dependencies

python -m venv .venv
.venv/bin/pip install --index-url https://download.pytorch.org/whl/cpu torch
.venv/bin/pip install sentence-transformers pytest
.venv/bin/python run_eval.py

.