FIFO Queue Webhook Deduplication: Idempotency Keys for Duplicate Events After 5 Minutes

Short answer: treat a five-minute FIFO dedupe window as a traffic-control measure, then enforce duplicate event handling with a durable idempotency key in the same transaction as the payment reconciliation result. The queue can reduce immediate repeats; it cannot decide whether a business effect already committed.

That distinction matters most in logistics, where a nightly reconciliation against a payment provider may update thousands of shipment records while operators are trying to recover from a partial run. A retry after a timeout, a replay from the provider, and a worker that committed just before losing its acknowledgment are different delivery attempts that can carry the same business event.

Timing is not identity.

What should a FIFO webhook queue do after a five-minute dedupe window?

Give each provider event a stable identifier and preserve it through the queue message, worker logs, database claim, and downstream request. Use the identifier as the idempotency key. The FIFO layer may suppress a rapid duplicate, but the consumer still needs a durable record for a duplicate delivered after five minutes, after a process restart, or after an operator requests a replay.

The invariant is narrow: one event ID may produce one committed reconciliation mutation. A process-local map cannot protect that invariant across workers. A read followed by an insert cannot protect it under concurrency either, because two workers can observe the missing row at the same time. A database uniqueness constraint makes the claim atomic.

The consumer should acknowledge only after the claim and the business mutation commit together. If the transaction fails, leaving the message available for retry is preferable to recording a successful claim without recording the reconciliation result. If the downstream payment API cannot join the transaction, use an outbox row carrying the same event ID and require the downstream operation to honor that key; this makes retries identifiable, but it does not magically make an HTTP call exactly once.

The recovery path for a delayed duplicate

Imagine a reconciliation run handling shipment SHP-4821. The worker claims event pay_7f3, writes the provider's settled status, and then loses its queue acknowledgment. The next delivery is not a new payment. It is another attempt to deliver the same event, and it must take the duplicate path without applying the status transition again.

The durable claim belongs beside the data it protects. A unique key on event_id rejects the second claim; the worker can then acknowledge that delivery because the original transaction already owns the business effect. Picture the nightly run as a sequence of small decisions rather than one giant batch: event pay_7f3 is claimed, shipment SHP-4821 is updated, the transaction commits, and only then does the worker acknowledge the delivery. If the acknowledgment disappears, the next worker sees the same event ID, loses the uniqueness race, and exits without touching the shipment row. If the database transaction fails instead, no durable claim remains and a retry can make the original attempt. That ordering is the recovery procedure, not an implementation detail, because an operator can replay one event and inspect one claim record without guessing whether a payment mutation happened. It also gives the SLO review a concrete question: how long can a replay arrive, and how much indexed state can the team retain for that period? Don't answer with the queue's timer alone. Five minutes is a queue behavior, not a universal retention policy.

This is also where SLOs become practical. Measure queue age, reconciliation completion latency, duplicate-claim rate, transaction failures, outbox age, and the time from alert to a replay-safe recovery. Capacity planning must include the indexed claim on every delivery, plus the extra load created by a payment provider retry storm. A FIFO queue may preserve order within its contract, but it does not remove database contention or downstream rate limits.

A small Go consumer that makes the boundary explicit

The question is often phrased as a Node.js implementation request, but the concurrency rule is language-independent. This Go handler shows the critical path: claim the event and write the reconciliation result in one transaction, then return success for a duplicate without repeating the mutation. The equivalent Node.js service should use its database driver's transaction API and the same unique constraint.

package main

import (
    "context"
    "database/sql"
    "encoding/json"
    "net/http"
)

type paymentEvent struct {
    ID       string `json:"id"`
    Shipment string `json:"shipment_id"`
    Status   string `json:"status"`
}

func reconcile(db *sql.DB) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if r.Method != http.MethodPost {
            http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
            return
        }
        defer r.Body.Close()

        var event paymentEvent
        if err := json.NewDecoder(r.Body).Decode(&event); err != nil || event.ID == "" {
            http.Error(w, "invalid event", http.StatusBadRequest)
            return
        }

        tx, err := db.BeginTx(context.Background(), nil)
        if err != nil {
            http.Error(w, "transaction unavailable", http.StatusInternalServerError)
            return
        }
        defer tx.Rollback()

        result, err := tx.ExecContext(r.Context(),
            "INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT (event_id) DO NOTHING",
            event.ID,
        )
        if err != nil {
            http.Error(w, "claim failed", http.StatusInternalServerError)
            return
        }
        claimed, err := result.RowsAffected()
        if err != nil {
            http.Error(w, "claim status unavailable", http.StatusInternalServerError)
            return
        }
        if claimed == 0 {
            w.WriteHeader(http.StatusNoContent)
            return
        }

        _, err = tx.ExecContext(r.Context(),
            "UPDATE shipments SET payment_status = $1 WHERE shipment_id = $2",
            event.Status, event.Shipment,
        )
        if err != nil {
            http.Error(w, "reconciliation write failed", http.StatusInternalServerError)
            return
        }
        if err := tx.Commit(); err != nil {
            http.Error(w, "commit failed", http.StatusInternalServerError)
            return
        }
        w.WriteHeader(http.StatusNoContent)
    }
}

The schema contract is more important than the handler's language: processed_events.event_id must be unique, and the shipment update must be safe for the event's business semantics. A status replacement is not automatically safe for every payment action; a capture, refund, or ledger posting may need an append-only record and a separate uniqueness rule.

Keep credentials and event identifiers out of ordinary application logs. Key material deserves explicit ownership, rotation, and access controls; the OWASP guidance is a useful baseline for that review. For rate-limited HTTP calls, honor Retry-After when the server sends it with a 429 response. Backoff changes pressure; it does not replace idempotency.

Which design fits the operational recovery requirement?

The comparison I use in capacity planning is about recovery ownership, not queue branding.

The catch is that a FIFO queue with a durable claim is not suitable when the job is really a multi-step workflow with compensation or fan-out and fan-in. Choose a workflow engine for that shape. Use a simpler queue when the work is an ordered delivery plus a transactional mutation, and document how an operator replays one event without replaying the entire nightly batch.

Your mileage may vary: the right retention period depends on the payment provider's replay behavior, the audit obligation, and the cost of keeping claim rows. I would not set it from the five-minute dedupe interval without evidence from those three inputs.

The useful design decision is therefore easy to state: let FIFO reduce duplicate work during a short retry burst, but let a durable idempotency key decide whether a reconciliation effect may commit. That rule survives delayed delivery, worker restarts, and an imperfect acknowledgment boundary.

References

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
• https://cheatsheetseries.owasp.org/cheatsheets/Key_Management_Cheat_Sheet.html