Code: Megapixel99/webCrawler
This project is a small search engine I wrote to understand how one works: a crawler, an inverted index in MongoDB, and a BM25 ranker, all written by hand instead of reaching for Elasticsearch.
The first version was not something you should point at the open web. It honored noindex and nofollow meta tags, but it ignored robots.txt entirely and had no rate limiting, which means it would hammer whichever host it happened to find.
So I fixed it. The parsing turned out to be the easy half. The interesting half was that politeness isn't really a parsing problem at all; it's a concurrency problem, and I got it wrong twice in ways that looked completely fine.
The rate limiter that wasn't
The crawler runs a pool of workers using Node's cluster module. The master process pulls URLs from a MongoDB frontier and forks a worker per URL:
var max_processes = Math.floor(os.freemem() / memUsed);
On my machine that's somewhere north of a hundred workers.
My first rate limiter was the obvious thing: a map of host to "next allowed time," checked before each fetch:
const nextAllowed = new Map();
if (Date.now() < nextAllowed.get(host)) return;
nextAllowed.set(host, Date.now() + DELAY);
This is wrong in a way that produces no error, no warning, and no symptom you'd notice locally. cluster workers are separate processes. They don't share memory. Every worker gets its own nextAllowed map, so every worker enforces a perfectly correct five-second delay against its own private counter, and the host on the other end receives a hundred requests in that window instead of one.
The insidious part is that each individual component is right. If you log the delay from inside a worker, it's five seconds. If you unit test the limiter, it passes. The bug only exists in the space between the processes, which is exactly the space that nothing in your test suite occupies.
Any per-process cache has this property. If you cache robots.txt in a module-level variable, you don't fetch it once per host, you fetch it once per host per worker.
Moving the state somewhere shared
The fix is that the limit has to live somewhere all the workers can see. I already had MongoDB as the shared frontier, so it became the shared politeness state too: a hosts collection with one document per host:
{
host: 'example.com',
robotsTxt: '...',
robotsCheckedAt: Date,
crawlDelay: 5,
nextAllowedAt: Date, // the rate limit slot
}
Which fixes visibility, and immediately introduces the second bug.
Check-then-act is not a check
Here's the natural way to use that document:
const host = await Host.findOne({ host, nextAllowedAt: { $lte: now } });
if (!host) return; // someone else is fetching
await Host.updateOne({ host }, { $set: { nextAllowedAt: now + delay } });
await fetch(url);
Read the state, decide, then write. Every worker now sees the same data, so this looks like it solves the problem.
It doesn't. Between the read and the write there's a window, and with a hundred workers starting simultaneously, all of them land in it. They all read nextAllowedAt in the past, all conclude they're clear, all write, and all fetch. Sharing the state didn't help, because the decision and the reservation were two separate operations.
Deciding and reserving have to be a single atomic step. In MongoDB that's findOneAndUpdate with the condition in the filter:
const claim = await Host.findOneAndUpdate(
{ host, nextAllowedAt: { $lte: now } },
{ $set: { nextAllowedAt: new Date(now.getTime() + delayMs) } },
{ new: true }
);
if (!claim) return DEFERRED; // another worker holds the slot
The database matches and updates the document in one operation. Exactly one worker's filter matches; everyone else gets null back and defers. The returned document is the permission; there's no second check to get wrong.
This shape isn't specific to Mongo. It's the same idea as SELECT ... FOR UPDATE, a Redis SET NX, or a compare-and-swap: don't ask whether you may proceed, ask for the thing itself and see whether you got it.
The deadlock I built while fixing that
There's a subtle trap in that filter. It only matches a document that already exists, and a host you've never crawled doesn't have one. So claim is null, the worker defers, nothing creates the record, and the host is never crawlable. Forever.
I fixed this the wrong way first, by adding upsert: true to the claim. That does create the missing document, but it changes what a lost race looks like: instead of null, a worker that loses now gets a duplicate-key error, because host is uniquely indexed. My error handler treated that as a failed crawl and deleted the URL from the frontier, so contention silently destroyed queued work.
What actually works is separating the two concerns. Seed unconditionally, then claim:
await Host.updateOne(
{ host },
{ $setOnInsert: { nextAllowedAt: new Date(0) } },
{ upsert: true }
);
$setOnInsert only writes on creation, so it can't clobber a live nextAllowedAt on a host that's mid-crawl. A new host starts at the epoch, which means immediately claimable. And with the seed separated out, null from the claim unambiguously means "throttled" rather than "throttled, or maybe this host has never been seen."
Fail closed, not open
One more decision worth being deliberate about: what do you do when you can't fetch robots.txt?
A 404 is an answer. It means "no rules," and you should cache that fact, otherwise you re-request a file that isn't there on every single page, which is its own kind of rude.
A timeout or a connection error is not an answer. You don't know what the rules are. It's tempting to proceed, because the alternative is doing nothing, but "I couldn't read your rules so I assumed they permit this" is not a defensible position. Mine returns three distinct results (a body, an empty string for 404, and null for unreachable) and null defers the URL to retry later rather than recording it as checked.
Testing the thing that matters
Most of this is invisible to the kind of test that asserts on return values. A rate limiter that reports "deferred" while still sending the request is exactly as rude as no rate limiter at all.
So the assertions that matter are about what the other side saw. I run a throwaway MongoDB and a local HTTP server standing in for a crawled host, and check the server's request log:
check('disallowed path reported', await nextQueue(url('/private/x')), DISALLOWED);
check('disallowed path never requested', hits.includes('/private/x'), false);
And for the concurrency, six workers against one brand-new host:
const racers = await Promise.all(
Array.from({ length: 6 }, () => nextQueue(url('/race'))));
check('exactly one worker won', racers.filter(r => r === CRAWLED).length, 1);
check('seed race made one doc', await Host.countDocuments({ host }), 1);
check('only one page request', hits.filter(h => h === '/race').length, 1);
That last assertion is the one I care about. Both of my broken versions would have passed a test that only checked return values.
What I'd take away from it
Politeness features have an unusual property: when they're broken, everything on your side still looks correct. There's no exception, no failing request, no degraded response. The only party who observes the bug is the person running the server you're hammering, and they experience it as your crawler being hostile.
That's a good argument for testing from the outside (asserting on what the remote server received rather than what your code returned) and for being suspicious of any correctness property that depends on state living in the right place. "Works on one process" and "works on a hundred" are different claims, and cluster makes it very easy to believe you've tested the second when you've only tested the first.
The crawler, the hand-written robots.txt parser and the BM25 ranker are all in the repo linked above. The two concurrency assertions are in test/politeness.test.js; they spin up a real MongoDB and a real HTTP server, so you can check the claim rather than take my word for it.
