How an in-process scheduler turned one nightly job into N duplicate writes, and why I moved scheduling out of the app entirely.
I was looking at staging data when I found two of something that should only ever exist once: two identical records for the same entity, same start date, created seconds apart. Not corrupted, not half-written. Just two, where there should have been one.
The job that creates those records runs once a night. So why did it run twice?
Because we run more than one copy of the service, and the scheduler lived inside the service.
The setup
A nightly job rolls every active entity forward into its next bounded time-window — one new record each, once a day. Standard NestJS:
@Injectable()
export class WindowGenerationService {
@Cron('0 8 * * *') // every day at 08:00
async generateNextWindows() {
const entities = await this.repo.findActiveEndingSoon();
for (const entity of entities) {
await this.repo.createNextWindow(entity);
}
}
}
This is the documented way to schedule work in NestJS. It's what you reach for. It worked perfectly the entire time we ran a single instance.
Then we scaled horizontally. Now there are three instances. @Cron doesn't know that. The decorator schedules the job in every process that boots it. Three instances, three timers, three simultaneous runs at 08:00 — and because there was no idempotency check and no lock, all three sailed past "does this already exist?" and each wrote its own copy.
Two things bothered me equally. One, duplicate data. Two, I was paying three containers to do the same work at the same time and then paying again to clean up the mess. The scheduler being in-process wasn't just a correctness bug, it was wasted compute by design.
What I considered
Option 1: a distributed lock. Keep @Cron, but have instances race for a lock — an advisory lock, or a key in a cache — and let the winner run. It works, and it trades a loud failure for a silent one. Duplicate writes are at least visible; you can find them and delete them. A missed run isn't: if the lock backend is down at 08:00, or an instance dies holding the lock and the TTL hasn't expired, the job just doesn't fire, and nobody notices until someone's window is missing days later. I'd be adding a whole new dependency, and a new way to fail quietly, to fix a problem I'd created by putting the timer in the wrong place.
Option 2: idempotency only. Add a check so a second run is a no-op. Cheap, correct, but it still wakes up three containers to do redundant work every night.
Option 3: take scheduling out of the app. The app shouldn't own when. Let an external scheduler own the clock and fire a single HTTP request at a normal endpoint. The app only owns what happens when it's poked. One trigger, one run, and horizontal scaling stops multiplying anything.
I went with 3, and kept the idempotency from 2 as a seatbelt.
The change
The @Cron decorator went away. The logic moved behind an endpoint:
@Post('jobs/run')
async runJob(@Body() body: RunJobDto) {
this.assertValidSecret(body.secret);
return this.jobs.run(body.jobKey);
}
An external scheduler fires this once at 08:00. The instance that receives it runs the job; the other two never hear about it.
The idempotency check stayed, because "fires once" is a promise infrastructure makes and occasionally breaks — retries, manual re-triggers, a restart mid-run:
async createNextWindow(entity: Entity) {
const existing = await this.repo.findByEntityIdAndStartDate(
entity.id,
entity.nextStartDate,
);
if (existing) return; // already done, no-op
await this.repo.create(/* ... */);
}
What it cost
The moment you turn a private nightly job into an HTTP endpoint, you've created a public button that runs a job. Anyone who finds it can hammer it. So the endpoint is guarded by a shared secret — and the comparison matters more than people expect:
private assertValidSecret(provided: string) {
const expected = this.config.cronSecret;
const a = Buffer.from(provided);
const b = Buffer.from(expected);
if (a.length !== b.length || !timingSafeEqual(a, b)) {
throw new UnauthorizedException();
}
}
A naive provided === expected leaks information through how long it takes to fail — a timing oracle you can walk character by character. timingSafeEqual compares in constant time. It's a small thing that reviewers usually don't flag; I'd rather not gamble that the endpoint stays undiscovered.
The other cost was dumber: picking the hour. The job has to run after everyone's start of the day, and "everyone" spans several US time zones — a time that's mid-morning in the east is pre-dawn out west. So the schedule is pinned to a fixed UTC hour chosen against the westernmost operating timezone, because that's the one that makes it safe everywhere. UTC in the crontab, human timezones in the requirement — the gap between those two is where the real decision lives.
The lesson
An in-process scheduler is a lie the moment you run more than one instance — it quietly multiplies by your replica count, and nothing in the code hints at it. Scheduling is a "when" concern, and it belongs outside the app; the app should only own "what happens." And the instant you expose a job as an endpoint to get there, it's a public endpoint — treat it like one, and keep the idempotency check anyway, because "it only fires once" is someone else's promise.
