Background jobs look simple right up until one of them dies silently in production and nobody notices for three days. A job that sends confirmation emails stops running. A job that syncs inventory data quietly falls behind. Nobody gets an error, because from the queue's perspective, nothing "crashed" — the job just failed and nobody was watching.
Most BullMQ tutorials stop at "job added, job processed." That's fine for a demo, but it's not what happens in a real system. In production, external APIs time out, workers restart mid-job, and retries without the right safeguards can make things worse, not better.
In this post, I'll skip the basic setup tutorial and goes straight into the patterns that actually matter: retries that don't cause a thundering herd, idempotency so retries don't duplicate side effects, dead-letter queues for jobs that keep failing, concurrency limits that protect your database, and how to catch jobs that are "done" but still stuck.
Quick Setup
If you haven't wired up BullMQ in a NestJS app yet, here's the minimum you need.
npm install @nestjs/bullmq bullmq ioredis
// app.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';
@Module({
imports: [
BullModule.forRoot({
connection: {
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT),
},
}),
BullModule.registerQueue({
name: 'notifications',
}),
],
})
export class AppModule {}
// notifications.processor.ts
import { Processor, WorkerHost } from '@nestjs/bullmq';
import { Job } from 'bullmq';
@Processor('notifications')
export class NotificationsProcessor extends WorkerHost {
async process(job: Job): Promise<void> {
// send the notification
}
}
That's the happy path. Now let's make it survive contact with production.
Retries with Exponential Backoff
The default instinct is to just add retries and move on:
await notificationsQueue.add('send-email', payload, {
attempts: 5,
});
The problem: without a backoff strategy, BullMQ retries as fast as it can. If the reason the job failed was a downstream API having a bad moment, five instant retries from every failed job in the queue can turn a blip into an outage — a thundering herd hitting a service that's already struggling.
Exponential backoff spaces retries out so the downstream system gets room to recover:
await notificationsQueue.add('send-email', payload, {
attempts: 5,
backoff: {
type: 'exponential',
delay: 1000, // 1s, 2s, 4s, 8s, 16s
},
});
For queues with high volume, add jitter on top of this. If a downstream outage causes thousands of jobs to fail at the same moment, exponential backoff alone still retries them all in near-lockstep — you've just delayed the thundering herd, not prevented it. A custom backoff strategy fixes that:
BullModule.registerQueue({
name: 'notifications',
defaultJobOptions: {
backoff: {
type: 'custom',
},
},
});
// custom-backoff.strategy.ts
export function customBackoffStrategy(attemptsMade: number): number {
const base = Math.min(1000 * 2 ** attemptsMade, 30000);
const jitter = Math.random() * 0.3 * base;
return base + jitter;
}
Register it on the worker's settings.backoffStrategy option so retries spread out instead of clustering.
Idempotency Check: Making Retries Safe
Retries assume it's safe to run the job again. That assumption breaks constantly. A job that charges a customer, sends an email, or writes to an external system can cause real damage if it runs twice — the first attempt may have actually succeeded downstream even though your worker crashed before it could confirm that success.
A naive fix is to check a local "already ran" flag before doing the work, then set it after:
async process(job: Job): Promise<void> {
const key = `send-email:${job.data.userId}:${job.data.templateId}`;
if (await this.idempotency.hasRun(key)) {
return; // already sent, retry is a no-op
}
await sendEmail(job.data);
await this.idempotency.markComplete(key);
}
This looks safe, but it has a gap: if sendEmail succeeds and then markComplete fails — a crash, a dropped connection, a database hiccup — the next retry has no record that the email already went out. It checks hasRun, finds nothing, and sends a second email. The idempotency check itself isn't idempotent.
The most reliable fix is to push the idempotency key to the provider you're calling, if it supports one. Some transactional email providers (like Resend, and similar) accept an idempotency key on the send request. Passing the same key on every retry lets the provider dedupe the send at the source — so even if your local markComplete never runs, a retry is a safe no-op regardless of what your own state thinks happened.
async process(job: Job): Promise<void> {
const key = `send-email:${job.data.userId}:${job.data.templateId}`;
await sendEmail(job.data, { idempotencyKey: key });
await this.idempotency.markComplete(key);
}
This removes the ambiguous window entirely for the retry path. If markComplete fails, sendEmail runs again on retry, but the provider silently no-ops it instead of sending a duplicate.
Not every downstream system supports this, though. When it doesn't, the safer pattern is to claim the key before doing the work, rather than checking and marking after:
async process(job: Job): Promise<void> {
const key = `send-email:${job.data.userId}:${job.data.templateId}`;
const status = await this.idempotency.getStatus(key);
if (status === 'completed') {
return; // confirmed already sent
}
if (status === 'pending') {
// a previous attempt claimed this key but never confirmed completion —
// this is the ambiguous case: it may or may not have actually sent.
// don't blindly resend; flag it for manual reconciliation.
throw new Error(`Ambiguous send state for${key} — needs manual check`);
}
await this.idempotency.claim(key); // insert with a unique constraint, status='pending'
await sendEmail(job.data);
await this.idempotency.markComplete(key); // status='completed'
}
This doesn't eliminate the ambiguous window — nothing fully can, without cooperation from the downstream system. What it does is make that window visible instead of silently resending: a job stuck in pending gets flagged for a human to check, rather than firing off a second email automatically. In most systems, an occasional job that needs a manual look is a far better trade-off than an automated retry that might double-send.
Dead-Letter Queues for Jobs That Keep Failing
Once a job exhausts its attempts, BullMQ marks it failed and moves on. If nothing is watching for that event, the job's failure disappears into the queue's history — it won't page anyone, and it won't show up unless someone happens to check.
A dead-letter queue (DLQ) gives failed jobs a place to land where they're visible and replayable.
// dead-letter.listener.ts
import { OnQueueEvent, QueueEventsListener, QueueEventsHost } from '@nestjs/bullmq';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
@QueueEventsListener('notifications')
export class NotificationsDeadLetterListener extends QueueEventsHost {
constructor(@InjectQueue('notifications-dlq') private dlq: Queue) {
super();
}
@OnQueueEvent('failed')
async onFailed({ jobId, failedReason }: { jobId: string; failedReason: string }) {
const job = await this.getJob(jobId);
if (job && job.attemptsMade >= job.opts.attempts!) {
await this.dlq.add('failed-notification', {
originalData: job.data,
failedReason,
failedAt: new Date().toISOString(),
});
}
}
}
From there, a small replay script lets you re-queue jobs from the DLQ once the underlying issue is fixed:
// replay-dlq.script.ts
async function replayDlq(dlq: Queue, notifications: Queue) {
const jobs = await dlq.getJobs(['waiting', 'delayed']);
for (const job of jobs) {
await notifications.add('send-email', job.data.originalData);
await job.remove();
}
}
The DLQ doesn't fix the underlying failure — it turns "silently lost job" into "visible, replayable job," which is the difference between a five-minute fix and a customer complaint three days later.
Concurrency Control and Backpressure
It's tempting to crank up concurrency to clear a backlog faster:
@Processor('notifications', { concurrency: 50 })
export class NotificationsProcessor extends WorkerHost {
// ...
}
Fifty concurrent jobs sounds fine until each one opens a database connection, and your Postgres pool only has 20 connections available. The queue isn't the bottleneck anymore — the database is, and now every other part of the app competes with the job queue for connections.
Size concurrency around your actual downstream constraints, not an arbitrary number:
@Processor('notifications', {
concurrency: 10, // matched to available DB pool headroom
})
export class NotificationsProcessor extends WorkerHost {
// ...
}
If the queue talks to a rate-limited third-party API, BullMQ's built-in limiter is a better fit than concurrency alone:
BullModule.registerQueue({
name: 'notifications',
limiter: {
max: 100,
duration: 60000, // max 100 jobs per minute
},
});
Detecting Stuck or Silently Hanging Jobs
The failure mode nobody plans for: a job that's technically still "active" from BullMQ's perspective, but is actually hung — stuck waiting on a call to an external service that never times out and never resolves. It won't show up as failed. It'll just sit there, occupying a worker slot indefinitely.
BullMQ's stalled-job detection catches part of this — if a worker dies mid-job without renewing its lock, the job gets marked stalled and retried:
BullModule.registerQueue({
name: 'notifications',
settings: {
stalledInterval: 30000,
maxStalledCount: 2,
},
});
That handles a crashed worker. It doesn't handle a worker that's still alive but stuck in a call with no timeout. For that, the job itself needs a hard timeout:
async process(job: Job): Promise<void> {
await Promise.race([
sendEmail(job.data),
new Promise((_, reject) =>
setTimeout(() => reject(new Error('Job exceeded timeout')), 15000),
),
]);
}
Without a timeout like this, "the job is still running" and "the job is stuck forever" look identical from the outside. This is usually where a team ends up building or buying observability into their job queues — once you've been paged for a queue that quietly backed up for six hours, you stop trusting "it'll show up as failed eventually."
Wrapping Up
None of these patterns are exotic — retries, idempotency, dead-letter queues, concurrency limits, and timeouts are well-known concepts. What's easy to miss is that BullMQ gives you the primitives, not the judgment calls: how long to back off, whether a job is safe to retry, what "stuck" actually means for your workload. Those decisions are what separate a queue that works in a demo from one that survives production traffic.
If you're setting up background jobs in NestJS for the first time, start with idempotency and a DLQ before you worry about tuning concurrency — those two alone catch the failure modes that actually wake people up at night.
