Your eval set is probably in your training set — here's how to check in ten minutes

You fine-tune a model, run your benchmark, and the score jumps six points.

Before you write that up, there's one question worth ten minutes: how many of those benchmark examples were in the training data?

If the answer is "some", part of that six points is a measurement of memory rather than capability — and there is no way to separate the two after the fact.

This is train/test contamination. It's one of the most common and least discussed reasons an offline number fails to reproduce in production, and it is almost never introduced deliberately.

How it gets in

Nobody copies their test set into training on purpose. It happens through ordinary steps:

• Merging public datasets. Two datasets that look unrelated often share a source. Instruction-tuning collections are especially prone to this — many are recombinations of the same handful of seed sets.
• Splitting after augmentation. Paraphrase or template-expand first, split second, and variants of one item land on both sides. The split looks random. It isn't.
• Re-scraping. Your eval set came from a site in March. Your training crawl hit the same site in June.
• Synthetic data from a model that saw the benchmark. You may be distilling memorised answers straight into your training file.
• Datasets that grow. Eval was frozen a year ago; train has been appended to weekly by three people since, and nobody re-checked.

The pattern: contamination arrives with pipeline changes. That's why a one-off audit doesn't stay true, and why this belongs in CI rather than in a notebook you ran once.

Why it's worse than the percentage suggests

If 5% of your eval set is contaminated and the model scores near-perfectly on that slice, it can move the headline number by several points — often the same magnitude as the improvement you're trying to demonstrate.

Worse, it biases decisions, not just reporting. You pick checkpoints, hyperparameters and data mixes by comparing eval scores. Contamination rewards whichever run memorised more, which is usually the run that trained longer on the contaminated subset. A genuinely worse model can outscore a better one.

And you can't correct for it afterwards by subtracting points, because you have no idea how the model would have done on those items unseen.

Three levels of overlap

Contamination detection usually gets framed as one number. It's really three questions, with three different costs and three different levels of confidence.

1. Exact

Join whichever fields define an example, compare byte for byte. A hash lookup: O(n), complete — no false positives, no false negatives, nothing to tune.

Always run it. If it finds something, you have a definite problem and there's nothing to argue about.

2. Normalized

Apply Unicode NFKC, lowercase, strip punctuation and symbols, collapse whitespace, then compare. Catches the same example after a reformat: smart quotes, a title-cased prompt, trailing whitespace, a markdown wrapper.

Still a hash lookup, still complete, still essentially free — and in practice it finds several times more matches than exact alone, because real pipelines reformat text constantly. Skipping this level is the single most common reason a contamination audit reports "clean" when it isn't.

3. Near-duplicate

The hard one. Two records are near-duplicates when they share most of their content but not all of it. The standard approach:

1. Turn each text into a set of shingles — overlapping word n-grams, typically 5-grams. (Character n-grams for texts shorter than the window.)
2. Define similarity as the Jaccard index of the two shingle sets: intersection over union.
3. All-pairs comparison is O(n²), so approximate it with MinHash: a fixed number of hash permutations (128 is common) reduce each set to a short signature whose agreement rate estimates Jaccard.
4. Group signatures into LSH bands. Two records become candidates if any band matches exactly.
5. Score every candidate on the shingle sets themselves, not on the signatures, and keep the pairs above your threshold. (Very long records are first reduced to a bottom-k sketch — the k smallest hashes — a uniform sample that stays stable when the record is edited.)

The detail worth internalising: the approximation lives in step 4, candidate generation — not in the signatures used for scoring. Near-duplicate detection can miss a small number of borderline pairs, but every similarity it reports is measured from the shingle sets rather than read off a MinHash signature — exact for ordinary-length records, and an unbiased bottom-k estimate for very long ones. A tool that reports signature agreement as its similarity is giving you a noisier answer than it needs to.

Picking a threshold

The Jaccard threshold is the only judgement call in the whole process. Rough orientation for word 5-grams:

Two adjustments that matter more than the table:

• Short records → higher threshold. Few shingles means each differing word costs a lot of similarity.
• Heavy boilerplate → strip it first. An identical system prompt on every row means you'll measure the template, not the content.

The reliable method is empirical: run at 0.8, read twenty borderline pairs, and move the threshold based on whether you would call them the same example. Five minutes, and it beats any rule of thumb — including this one.

How to report it

Report contamination rate as the fraction of eval records with at least one match in train. Records, not pairs: a training file that duplicates one eval example forty times is one contaminated eval record.

And report per level:

exact:      31 eval records
normalized: 12 eval records
near:       27 eval records   (threshold 0.8)
-----------------------------------------
70 of 1,000 eval records (7.00%)

The first two lines are certainties. The third depends on a threshold you chose and a reviewer is entitled to argue with it. Keeping them separate is what makes the number defensible.

Checking your own files

I built SplitCheck because I was tired of writing this script slightly differently every time. Drop a train file and an eval file on the page and it runs all three levels in a Web Worker — nothing is uploaded, there's no upload endpoint in the app, and the worker bundle contains no network APIs at all. You get the per-level breakdown, the matched pairs with both snippets, and your training file with the overlap removed.

It's free and there's no sign-in. There's also a $29 CLI for data that can't go in a browser and for CI, but the browser version isn't crippled to sell it — same three levels, same code.

If you'd rather build it yourself with datasketch or text-dedup, do that. It's an afternoon, the libraries are good, and you'll understand your own pipeline better for it. What you're buying, if you buy anything, is the afternoon.

What to do when you find overlap

The instinct is to delete from the eval set. Resist it. Removing eval examples changes what your benchmark measures and breaks comparability with every number you've already published.

Clean the training set instead: remove every train record that matched, retrain, re-evaluate. Your eval set stays fixed, historical comparisons stay valid, and the new number is honest.

Then:

• Keep the report next to the model artifacts. In six months, when someone asks whether the eval was clean, you want a file rather than a memory.
• Compare the contaminated and clean scores. That delta is the most useful diagnostic you'll get all week.
• Review the borderline pairs before deleting. At 0.8 there will be a few genuinely distinct examples that share phrasing.

What no lexical method will catch

Being clear about the ceiling matters more than the pitch:

• Semantic paraphrase sharing no 5-gram. That needs embedding similarity — slower, less interpretable, non-deterministic across model versions, and it will also flag pairs that merely share a topic.
• Contamination through a third model. If your training data came from a model that memorised the benchmark, the leak is in weights, not in matching text.
• Fields you didn't compare. Overlap in a column you excluded is invisible.
• A base model's pretraining corpus. Nothing here can see that. The honest response is to say so when you report the score.

None of that makes the check less worth running. Exact and normalized overlap is common, entirely detectable and completely fixable — and it takes ten minutes to find out.

Disclosure: this post was drafted with AI assistance and reviewed before publishing.