What a Rate Limiter Reports, and What the Server Receives

Code: Megapixel99/webCrawler

I have a small search engine whose crawler runs as several Node processes under the cluster module, and its rate limiter used to be per-process. Each worker enforced a perfect five-second delay against its own state, so six workers meant six independent limiters and the host on the other end received six requests at once. The fix was an atomic claim in the shared database, and the test that caught it works by asserting on what the server received rather than on what the limiter reported.

That test is more reusable than the limiter it was written for, so I pointed it at Bottleneck, which is the most-downloaded rate limiter on npm at about 8.2 million installs a week.

With its default settings, under cluster, six workers configured for two requests per second delivered 14.04 requests per second, and every one of those limiters reported that it was spacing its jobs 495.8 ms apart.

What the library actually promises

Worth establishing first, because the headline sounds like an accusation and is not one.

Bottleneck's default is datastore: "local", and LocalDatastore keeps _nextRequest, _running and _done as ordinary instance fields that __submit__ reads directly. There is no inter-process coordination anywhere in the library: I grepped the source for process.send, worker_threads, Atomics and SharedArrayBuffer and found none of them, and every occurrence of the word "cluster" in that codebase refers to Redis Cluster rather than to Node's cluster module.

The README says clustering works across multiple Node instances using Redis, which is true. So local means local, exactly as documented; a limiter that coordinates nothing across processes is behaving correctly when it fails to coordinate across processes.

The finding is not that the library is broken. It is that the default configuration degrades silently under cluster, and that the limiter's self-report is identical in the working and broken cases.

The harness

A primary process starts an HTTP server on a random port and records process.hrtime.bigint() for every arrival. It forks N workers, and each one builds a single Bottleneck with minTime: 500 and maxConcurrent: 1, then schedules six GETs through it. Each worker also reports the gaps between its own job starts, which is the limiter's own view, so both numbers come out of the same run and can be compared directly.

The verdict is the smallest gap between consecutive arrivals at the server. Nothing in it consults the limiter.

The result

Three repeats of the failing arm gave 30, 30 and 31 violations out of 35; it is not a marginal effect. The aggregate rate also tracks the worker count. That is the signature of N independent limiters rather than of one limiter behaving badly:

The ratios run slightly above the worker count because the workers do not start in lockstep, so the window from first arrival to last is a little shorter than any single worker's own span. The shape is what matters: add a worker, add a worker's worth of traffic.

The two controls are the point

The single-worker arm exists to show the harness measures spacing correctly when spacing is actually happening. The ioredis arm exists to show it passes a correctly coordinated limiter across the same six workers.

Without both of them, "0.1 ms" is indistinguishable from a broken test. That is not a hypothetical concern here: the original bug in my own crawler survived precisely because the only thing checking the limiter was the limiter, and a measurement that can only produce the answer you expect is not a measurement. If I am going to publish a number about somebody else's library, the least I can do is demonstrate that my instrument can tell the two outcomes apart.

The ioredis arm also settles what the failing arm means. Six workers coordinating through Redis hold 2.00 requests per second exactly, so the 14.04 is not an artefact of cluster, of the harness, or of six processes contending for a socket. It is the absence of shared state.

Why nothing caught this

Bottleneck has fifteen test files and not one of them contains createServer or .listen(. Every assertion in the suite is made against the limiter's own timing.

That is a reasonable way to test a scheduler and it cannot express the failure above, because in every arm of my sweep the limiter's internal timing was correct. Each worker really did space its own jobs 495.8 ms apart, and no value any worker could have reported would have revealed the problem; no worker had the information: the fact being measured only exists in the aggregate, at the other end of a socket.

The last commit on Bottleneck is dated 2020-07-21, so this is a note for people using it rather than a defect report expecting a fix. robots-parser, which is the other half of a polite crawler, does no scheduling at all: it exposes getCrawlDelay() and contains no setTimeout or queue of any kind. So the path from "robots.txt asks for ten seconds" to "ten seconds actually enforced across workers" is not covered by either package, and if you are assembling politeness out of those two parts, the joint is yours to test.

What generalises

If a component's job is to affect something outside itself, the component's own opinion of whether it worked is worth nothing, and it will be reported in the same confident tone whether it worked or not.

The test that follows from that is cheap. Put something at the far end that has no reason to cooperate, record what it received, and compare. Fifty lines here, most of them the HTTP server; then break it deliberately, so you know the recording end can tell the difference.

For a rate limiter the practical version is short. If you run under cluster, pm2 in cluster mode, or multiple containers, and your limiter has no shared datastore configured, then your configured rate is per process. Your actual rate is that number times however many processes you happen to be running. Bottleneck will tell you the interval is correct. It is, in the only place the library can see.

The harness, both controls and the sweep are in bottleneck-cluster/ in the crawler's repo, and each arm is one command. Node v24.11.1, Bottleneck 2.19.5, macOS, Redis 7 in Docker.