Expiring Edtech Holds: A Node.js Webhook Queue with Idempotent Backoff

Short answer: for an edtech reservation that expires after a fixed hold window, use a durable webhook queue with an explicit retry state, exponential backoff, a dead-letter queue (DLQ), and manual redrive; make the delivery ID idempotent before tuning worker capacity.

The production scenario I would design around is a seat held for 10 minutes while a learner completes payment. When the hold expires, the reservation service emits a webhook to the enrollment system. A transient timeout should wait and retry. A malformed contract or a revoked credential should stop consuming worker capacity and become an operational decision. That distinction is the useful boundary.

What should a Node.js webhook retry queue do with failed deliveries?

The queue should carry an envelope, not just a payload: a stable delivery ID, reservation ID, event type, creation time, attempt number, and the next eligible time. The worker claims one envelope, verifies that the hold-expiry action is still valid, sends the webhook, and acknowledges only after the receiver has returned an acceptable response. A timeout is not proof that the receiver did nothing, so the receiver must treat the delivery ID as an idempotency key.

That last sentence is the part teams skip. If the receiver commits enrollment and the response disappears, a retry can arrive after the side effect already exists. A unique constraint or an idempotency record keyed by delivery ID turns that duplicate into a read of prior work instead of a second enrollment. HMAC is a reasonable way to authenticate the body and protect its integrity; RFC 2104 defines the keyed-hash construction, but it does not define replay protection, so include a timestamp and reject an envelope outside the receiver's allowed clock window.

Consider the payment boundary in this reservation flow. The expiry worker sends reservation.expired with delivery ID hold-781:expired, the enrollment service marks the seat available, and then the connection drops before the worker sees the response. The worker must enqueue the same ID, not mint hold-781:expired:retry-1 and hope the receiver can recognize the relationship. On the second delivery, the enrollment service should verify the signature, find the completed operation under the original ID, return the same outcome, and avoid changing seat state again. If the first attempt timed out before the receiver committed anything, the same ID should still be safe to process once. That is why idempotency belongs in the contract and in the data model, with an explicit retention period long enough to cover the maximum retry and redrive window; a queue policy cannot manufacture that guarantee after the fact.

The queue is at-least-once in the design sense: duplicates are possible, and correctness belongs at the application boundary. Node.js can implement the same state machine as the Go example below. The language is incidental; the persisted attempt and identity fields are not.

How should exponential backoff, failed webhooks, and a DLQ interact?

Retry policy needs two decisions: what is transient, and when is automatic recovery no longer responsible. Retry network timeouts and temporary overload with a capped exponential delay plus jitter. Do not retry a deterministic schema rejection forever. After the attempt budget is exhausted, publish the unchanged envelope to a DLQ and acknowledge the source item so one poison message cannot monopolize the consumer.

Here is the policy core. It deliberately returns an action instead of performing queue I/O, which lets tests cover the boundary cases without a broker.

package main

import "math/rand"

type Outcome string

const (
    Retry  Outcome = "retry"
    Dead   Outcome = "dead-letter"
    Accept Outcome = "ack"
)

type Delivery struct {
    ID      string
    Attempt int
}

type Decision struct {
    Outcome Outcome
    Attempt int
    Delay   int
}

func decide(d Delivery, temporary bool, maxAttempts, baseDelay, capDelay int) Decision {
    if !temporary {
        return Decision{Outcome: Dead, Attempt: d.Attempt}
    }

    next := d.Attempt + 1
    if next > maxAttempts {
        return Decision{Outcome: Dead, Attempt: next}
    }

    delay := baseDelay
    for i := 1; i < next; i++ {
        if delay >= capDelay/2 {
            delay = capDelay
            break
        }
        delay *= 2
    }
    if delay > capDelay {
        delay = capDelay
    }
    return Decision{Outcome: Retry, Attempt: next, Delay: delay + rand.Intn(baseDelay)}
}

func main() {}

In real code, inject the random source and test the delay bounds rather than asserting one jitter value. Also decide whether maxAttempts means total deliveries or retries after the first delivery; the example uses total deliveries. An off-by-one here changes the error budget quietly.

The retry queue must not extend the business hold. If the learner's reservation expired at 10:00, a webhook arriving at 10:12 should carry the event time and the receiver should apply the domain rule, not infer freshness from arrival time. That prevents a delayed message from reopening a seat or charging a second payment. The queue is a delivery mechanism, not the source of truth.

When is DLQ redrive safer than another automatic retry?

Redrive is safe only after someone can name the failure class and the expected result. Inspect a sample of DLQ envelopes, group them by destination and error category, fix the receiver contract or credential, then release a bounded batch. Watch success rate, oldest message age, retry volume, and the receiver's idempotency conflicts while that batch drains.

Start small.

An automatic redrive of every dead letter is just a retry storm with a more reassuring noun. It can also erase the evidence needed to distinguish a bad payload from a downstream outage. Keep the original attempt count and an operator-generated redrive ID in the new envelope; that gives the audit trail enough shape to answer “what changed?” without pretending the queue is a permanent event store.

The catch is that this design is not suitable when delivery is one state in a long, branching business workflow. If the process needs joins, compensating actions, human approval, or durable timers spanning many stages, use a workflow engine or an application-owned state machine. Stick with a simpler queue when the job is a bounded handoff and the team can own a clear runbook. More machinery does not improve a webhook whose real problem is a missing idempotency contract.

What should a platform team buy, build, and measure for webhook recovery?

I use the following buy-versus-build table during capacity planning. The “buy” column means adopting a managed queue or workflow service; “build” means operating the queue, workers, storage, and replay controls yourself. Neither column removes the receiver contract.

Measure the service against a delivery SLO, not a vague promise that “the queue is healthy.” For example, define the maximum age for a successful reservation-expiry notification, the acceptable duplicate rate after receiver idempotency, and the time to quarantine a poison message. Then size consumers from arrival rate, average processing time, retry amplification, and the recovery rate of the destination. A queue that is empty because workers are failing fast is not healthy.

My decision rule is narrow: preserve identity first, classify failures second, and add capacity only after the retry graph shows that capacity is the constraint. Your mileage may vary if the receiver contract is outside your team's control; in that case, the strongest improvement may be a reconciliation job that compares reservation truth with enrollment state.

References

• https://www.rfc-editor.org/rfc/rfc2104
• https://github.com/sidekiq/sidekiq/wiki