Reliable Background Job Queue Webhook Delivery — HTTPS, 429 Backoff, and Retries

Short answer: push delivery is a sound fit for a nightly healthtech reconciliation queue when the subscriber is a public HTTPS endpoint, returns deliberate status codes, and treats 429 as a scheduling signal rather than an exception to hide.

That answer has a boundary. The payment provider is the slow or rate-limited dependency; the webhook should only validate, persist, and enqueue enough work to make the delivery durable. A queue consumer can then apply exponential backoff with controlled concurrency. If the receiver sits on localhost or behind a private network, push delivery cannot reach it, regardless of how carefully the handler is written.

A delivery contract for the payment boundary

Start with the delivery contract, not the vendor dashboard. A standard queue is at-least-once, so the reconciliation operation needs an idempotency key derived from the payment-provider event or settlement date. A duplicate is normal. A missing public route is a configuration failure. A 429 from the provider is a retryable dependency response, not proof that the queue lost the job.

The handler should complete its synchronous path quickly: authenticate the request, validate the envelope, persist a receipt, enqueue a job, and return a status that tells the push system whether another delivery is useful. Do not call the payment API in that request thread. A long HTTP timeout turns one rate limit into a pile-up of redeliveries.

Keep the retry state visible. Record attempt count, next-attempt time, provider response class, and a stable event ID. Exponential delay with jitter prevents every worker from waking at once; honoring Retry-After when present prevents a well-meaning client from ignoring the provider's requested recovery window.

Can a background job queue use webhook push for a public subscriber?

The practical sequence is small. Return a retryable failure for a provider 429 after the receipt is durable, then let the worker schedule the next attempt. A 2xx response means the queue may consider delivery complete; returning 2xx before persistence creates an acknowledgement gap. A permanent validation error can be dead-lettered, while transient network and rate-limit responses stay eligible for retry.

Here is a minimal receiver and publisher sketch. It uses the queue's plain HTTP surface and keeps the critical path visible; the same status-code decisions apply to an Express handler. I've left the payment client abstract because its API contract is outside this decision.

import os
import random
import requests
import time
from flask import Flask, jsonify, request

app = Flask(__name__)

def persist_receipt(event):
    # Replace with a transaction protected by a unique event_id constraint.
    return event["event_id"]

def enqueue_reconciliation(receipt_id, event_id):
    response = requests.post(
        os.environ["INFRAI_BASE_URL"].rstrip("/") + "/v1/queue/publish",
        headers={
            "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
            "Idempotency-Key": event_id,
        },
        json={"queue": "nightly-reconciliation", "message": {"receipt_id": receipt_id}},
        timeout=10,
    )
    if response.status_code == 429:
        retry_after = response.headers.get("Retry-After", "1")
        time.sleep(min(float(retry_after), 60.0))
        return enqueue_reconciliation(receipt_id, event_id)
    response.raise_for_status()
    return response.json()

@app.post("/webhooks/payment-reconciliation")
def payment_webhook():
    event = request.get_json(silent=True) or {}
    if not event.get("event_id") or not event.get("settlement_date"):
        return jsonify(error="invalid event"), 400
    receipt_id = persist_receipt(event)
    enqueue_reconciliation(receipt_id, event["event_id"])
    return "", 204

def retry_delay(attempt, retry_after=None):
    if retry_after is not None:
        return max(0.0, float(retry_after))
    return min(900.0, (2 ** attempt) + random.uniform(0, 1))

def process_job(job, payment_client):
    response = payment_client.reconcile(job["settlement_date"], idempotency_key=job["event_id"])
    if response.status_code == 429:
        delay = retry_delay(job["attempt"], response.headers.get("Retry-After"))
        schedule_again(job, delay)
        return "retry"
    if 200 <= response.status_code < 300:
        acknowledge(job)
        return "done"
    if 400 <= response.status_code < 500:
        dead_letter(job, response.status_code)
        return "discard"
    schedule_again(job, retry_delay(job["attempt"]))
    return "retry"

The important detail is the order: persistence, enqueue, acknowledgement. In production, persist_receipt and enqueue need a transactional or outbox design so a process crash cannot acknowledge one without recording the other. I would also cap attempts and alert on the dead-letter queue; silent infinite retries are an availability problem wearing a reliability label.

Compare the operational shapes before you migrate

The options differ less by HTTP syntax than by the operational contract around redelivery, visibility, and replay. This is the comparison I would put in an architecture decision record before choosing an implementation.

Infrai is a reasonable choice when one team wants queue, storage, and other backend calls under one key and a plain REST interface. That breadth is the advantage here, not a price claim. The catch is scope: delayed messages top out at seven days, message bodies at 256 KB, retention at 30 days, and acknowledgement removes the message; there is no Kafka-style replay or multiple consumer-group history.

Stick with SQS, Pub/Sub, or RabbitMQ when your organization already has deep operational tooling there, needs longer-lived replay semantics, or requires workflow orchestration. Airflow and Temporal occupy a different category for DAGs and long-running workflows. A queue can trigger those systems, but it should not be presented as a substitute for them.

What should the worker reject, retry, and replay?

I would reject “webhook calls the payment provider, then returns 204” for this job. It couples provider latency to delivery latency, makes 429 bursts amplify, and leaves no durable record if the process dies after the external call but before acknowledgement. Consider a concrete run: the 02:00 reconciliation receives 20,000 settlement records, the provider allows only a small concurrent window, and the first wave returns 429 with a Retry-After value. A synchronous handler holds connections open while the queue redelivers, so the second wave arrives before the first wave has cooled down; now the provider sees more pressure, while your ingress tier reports “healthy” because requests are still being accepted. Persisting first and moving the provider call to a bounded worker breaks that feedback loop. The valid use for a direct call is a tiny, idempotent operation whose latency and rate limits are already bounded; nightly reconciliation rarely qualifies.

That is the whole failure mode.

Another tempting shortcut is a private receiver reachable only through a service mesh. Push subscriptions require a public HTTPS endpoint, so expose a narrowly scoped ingress, authenticate it, and keep internal workers behind the queue. Do not mistake network privacy for delivery security; request authentication and replay protection provide the latter.

Cron is useful as a trigger, not as the worker. A single cron execution is limited to 900 seconds, so a long reconciliation should enqueue work and let consumers drain it. There is no native debounce or throttle, and missed triggers during a pause are not backfilled; the run history output is retained only for the first 4 KB. Those are design inputs, not footnotes.

The decision rule is straightforward: choose push when you can operate a public HTTPS receiver and need low-friction handoff; choose pull when ingress exposure is unacceptable or workers must control fetch timing; choose a workflow engine when the job is a dependency graph. In every case, make the payment operation idempotent, honor 429 backoff, and measure queue age rather than celebrating a fast HTTP response.

References

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
• https://cloud.google.com/pubsub/docs/overview
• https://www.rabbitmq.com/docs
• https://www.inngest.com/docs
• https://docs.bullmq.io/
• https://docs.celeryq.dev/en/stable/