I built a research agent that has to quote its sources, and a gate — plain code, no model involved — that checks every quote actually appears in the page it cites.
Then I ran 20 questions through it and the gate struck out 22 of 47 claims. Nearly half.
I was pleased with myself for about ten minutes.
The rule
Every claim the agent makes needs two things: a sentence copied out of a real page, and the link it came from. Then deterministic code checks two conditions. Is that link one we actually fetched? And is that quote present in those exact bytes?
No to either, and the claim gets struck out of the report. That gate is the whole product — the model proposes, code disposes.
The security line is one array:
const ALLOW = ['en.wikipedia.org', 'developer.mozilla.org',
'docs.python.org', 'datatracker.ietf.org'];
The model suggests sources. It never fetches anything. Code decides what may be requested, so a suggested URL pointing somewhere interesting is simply dropped before a request goes out.
The part where I check my own work
I have a rule on this project: no number goes on screen until I know exactly what it means. So before celebrating a 47% rejection rate, I pulled all 18 quote-rejections and searched for them by hand in the cached pages the gate had judged them against.
Twelve of the eighteen were there. Character for character. The model had copied them perfectly and my gate called them missing.
The bug, which is embarrassing
My HTML-to-text step turned every tag into a space. So a page that reads:
Doppler, who described the phenomenon
became, in my cached copy:
Doppler , who described the phenomenon
The model quoted the page as a human reads it. I compared that against my own corrupted transcript. Every footnote marker did the same thing: [1] became [ 1 ].
It got worse. Wikipedia embeds JSON inside HTML attributes, and that JSON contains > characters. My tag-stripping regex <[^>]+> stops at the first > it finds — which was inside the attribute — so raw template code leaked into the middle of sentences.
Three repairs to the extractor: drop attribute values before stripping tags, remove inline tags with no separator space at all, and cut the raw TeX Wikipedia leaves inside its formulas. Then the comparison learned to ignore a space before a comma and padded footnote brackets.
Proving the fix didn't just go soft
This is the part I'd skip if I were being lazy, and it's the part that matters. A checker that passes everything is not a fixed checker.
So I ran the repaired gate against deliberately altered quotes:
| mutation | result |
|---|---|
| verbatim control | PASS |
| one digit changed | REJECT |
| negation inserted | REJECT |
| a word dropped | REJECT |
| a noun swapped | REJECT |
| a sentence invented | REJECT |
Zero of eighteen matched before the fix. Eighteen of eighteen after — and it still rejects every mutation. It didn't buy its pass rate by lowering the bar.
Re-run, measured fresh
Same 20 questions, same model. 53 claims, 45 verified, 8 rejected. Quote failures dropped from 18 to 5. Thirty-seven API calls, $0.037 for the batch.
And now the remaining rejections are worth reading. One is a quote the model assembled itself that appears nowhere on the page. One is a real quote my 8,000-character window never saw — it starts 67 characters past the cut, so that one is still my fault. One is the model ending a sentence with a period where the page has a comma.
The failure the gate never saw
All of this was downstream. Upstream, the agent was inventing sources.
Of the URLs it proposed in that batch, ten came back dead. From an earlier run I checked seven of those dead links one by one — against Wikipedia's own page logs, Python's git history, and the Internet Archive. At least six were paths that had never existed on those sites.
A Wikipedia article called "Variable declarator". A python.org page called "history". They read perfectly, which is exactly why nobody clicks them.
One more was a real Wikipedia link with the apostrophe dropped. That one isn't invention, it's a typo, and calling it a hallucination would be the same sloppiness I'm complaining about.
What I'd take from this
The model invented some of its sources, made up a handful of quotes, and copied the rest honestly. My checker punished twelve of the honest ones and never looked upstream at all.
A verifier that fails on correct behaviour is worse than no verifier. If its rejections ever feed back into how the model picks quotes — a retry prompt, a fine-tune, a "you got this wrong" — you have just trained it away from copying accurately.
Whatever you put downstream of a model, test it against known-good inputs before you trust a single one of its refusals.
What this still doesn't do
It reads four documentation domains only; a narrow allowlist is what makes the check meaningful. The gate proves a quote is real, not that it answers your question — one claim passed with a genuine quote attached to the wrong RFC. Three of the dead links are still unadjudicated, so six invented sources is a floor, not a total. And the sources that loaded fine were never tested for whether they support their claims at all.
Run it yourself
Workflow, questions, and the audit script that produced the mutation table:
👉 https://github.com/Ships-Itself/builds
I build one of these on camera each week and publish the real numbers, including the ones that make me look bad. The video version of this is on the Ships Itself channel.
