Short answer: Use cron while one process can finish scheduled cleanup before the next sweep and a failed run can safely wait; choose a durable background job queue once logistics webhooks need independent retries, restart recovery, and a dead-letter state.
Schedule discovery, not delivery. Let a periodic scanner find due records in durable storage, then let workers claim one record at a time with an idempotency key. Keep cron-only execution for small cleanup batches whose whole-run retry is acceptable.
This distinction matters because scheduled cleanup and an outbound webhook have different failure boundaries. Deleting expired delivery-attempt records is usually a set operation: run it again with the same cutoff and the result converges. Sending shipment.delayed to a carrier is an effect outside your database. A timeout does not tell you whether the carrier committed that effect. Retrying may be correct, or it may produce a second delivery.
No scheduler can infer which one happened.
Govern retries through the delivery record
Write the invariant before choosing the mechanism: for each (tenant_id, shipment_id, event_type, event_version), the system records one logical delivery, permits multiple transport attempts, and never runs two active attempts concurrently. Call that at-least-once transport with idempotent processing, not exactly-once delivery. The network cannot make the stronger promise on its own.
The database record is the authority. It needs a stable idempotency key, a payload reference or immutable payload, next_attempt_at, an attempt count, a claim owner, a lease expiry, and a terminal state. Put a unique constraint on the logical key. The queue message, if there is one, should carry the record identifier rather than become the only copy of business state. That choice makes reconciliation possible when publishing and committing cannot be one atomic operation.
A lease closes a specific crash window. A worker atomically changes a due record from pending to leased only when no live lease exists. If the process exits after claiming but before recording the result, the lease expires and another worker can try later. The recipient still needs to deduplicate by the stable key because a process can also exit after the remote endpoint accepts the request but before the local success commit.
Leases expire. Evidence should not.
Here is a storage-oriented sketch. It is Python pseudocode because the transaction boundary is the point; the same compare-and-set belongs in the persistence adapter used by a Node.js worker.
def claim_due_delivery(db, worker_id, now, lease_until):
return db.fetch_one(
"""
update webhook_deliveries
set state = 'leased',
lease_owner = :worker_id,
lease_expires_at = :lease_until
where id = (
select id
from webhook_deliveries
where state = 'pending'
and next_attempt_at <= :now
and (lease_expires_at is null or lease_expires_at < :now)
order by next_attempt_at, id
for update skip locked
limit 1
)
returning id, destination, payload_ref, idempotency_key, attempt_count
""",
worker_id=worker_id,
lease_until=lease_until,
now=now,
)
Do not copy that query blindly: transaction syntax and lock behavior vary by database. The required property is narrower and testable — two workers racing for one due record cannot both receive it. I'm not sure a generic ORM abstraction can preserve that property across every supported database; a concurrency test against the actual engine resolves the uncertainty.
How should a simple Node.js background job queue schedule cleanup retries and dead letters?
Use two clocks. A low-frequency scanner selects due delivery records and enqueues their identifiers, while workers own per-record retries. A separate cleanup sweep deletes or archives terminal attempt records according to the retention policy. This keeps retention cleanup from competing conceptually with webhook redelivery, even if both begin from scheduled work.
The retry policy should classify outcomes, not repeat everything. Treat an explicit success as terminal. A 429 can be retried after honoring a valid Retry-After policy; a connection timeout is ambiguous and therefore retryable only with the same idempotency key. Most permanent client rejections should stop rather than consume the entire retry budget. Use capped exponential backoff with jitter so a carrier recovery does not release every delayed shipment event at once. The exact cap and attempt count are workload decisions: set them from the recipient's recovery expectations, your delivery deadline, and the rate at which operators can inspect terminal failures.
Dead-letter is a state, not a trash can. Preserve the logical key, destination identifier, attempt history, sanitized failure classification, and payload reference. Do not store credentials in the record, and do not permit arbitrary destinations supplied by a tenant to bypass outbound controls. Webhook delivery is an SSRF boundary — allowlist destinations when the business case permits it; otherwise validate input and enforce network-layer restrictions. DNS resolution and redirect handling belong in that threat model.
For replay, create a new attempt generation under the same logical idempotency key, record who requested it, and require an explicit reason. A replay button that silently resets attempt_count destroys evidence. It also makes a poison payload look like a transient outage.
Comparing cron and workers at the ambiguous timeout
The choice is less about throughput than recovery granularity. A cron-only process can be quite good when cleanup is idempotent, bounded, observable, and allowed to wait for the next interval. It becomes awkward when one bad destination blocks unrelated records or when operators need to retry one shipment event without rerunning the batch.
| Constraint | Cron sweep | Durable worker queue |
|---|---|---|
| Recovery unit | Usually the batch or cursor range | One delivery record |
| Retry timing | Next sweep unless custom state is added | Per-record due time |
| Crash recovery | Rerun an idempotent sweep | Expire a lease and reclaim |
| Poison work | Must be modeled explicitly | Terminal state or dead-letter path |
| Operational load | Fewer moving parts | Queue health, leases, lag, and replay controls |
| Best fit | Small, bounded retention cleanup | Independent webhook delivery with mixed outcomes |
Operational cost follows recovery granularity
The catch is real: a durable queue adds another control plane, more metrics, and reconciliation work. It is not suitable when a nightly deletion query completes comfortably inside its window, reruns safely, and has no per-item delivery deadline. Stick with a cron sweep in that case. Conversely, don't mistake a hosted scheduler for a delivery engine. Scheduled workflows can be delayed during high load, and queued work may not start at the requested minute, so a repository workflow scheduler is useful for noncritical maintenance triggers but a poor authority for time-sensitive webhook retry state.
There is also a middle option: use the primary database as the durable work table and run a small worker pool. This avoids introducing a separate broker, but it transfers queue concerns to database design. Polling cadence, indexes, row-lock contention, lease repair, retention, and replica lag become your problem. For moderate workloads where the team already operates the database well, that can be the simplest worker queue. For high fan-out or strict isolation between tenants, a dedicated broker may justify its operational cost. Your mileage may vary.
Rollout without changing the recipient contract
Test the windows that diagrams omit. Pause a worker immediately before the outbound call, immediately after the recipient accepts it, and immediately before the local success commit. Run two claimers against the same row. Advance the clock past a lease. Return a sequence such as 429, timeout, then success, and assert that every request carries the same idempotency key. Feed a permanent rejection and verify that it stops. Finally, replay a dead-letter record and confirm that the audit trail remains intact.
Break it on purpose.
Observe four separate things: due-work lag, active leases older than expected, attempts by outcome class, and terminal records awaiting review. Queue depth alone is weak evidence; it cannot distinguish healthy future work from a stuck lease. Alert on breached delivery deadlines and oldest-record age, then use counts for diagnosis. Keep destination hostnames and tenant identifiers out of high-cardinality metric labels; attach them to controlled logs or traces instead.
Roll out compactly. First, add durable delivery records and idempotency keys while the existing scheduler still drives execution. Next, make processing claim-based and run one worker with outbound sending disabled in a shadow environment, comparing claims with the old selection logic. Then enable a small tenant cohort, verify duplicate suppression at the recipient contract, and expand gradually. Only after redelivery is independent should the old batch sender be removed. Keep the scheduled retention cleanup; it still has a job, just not the job of proving delivery.
The decision rule is blunt: keep cron for convergent cleanup, and introduce durable workers for independently recoverable effects. More machinery earns its place only when the failure boundary demands it.
