What is a webhook? How webhooks work, and the six ways they break in production

This is a cross-post. The original on the Railhook blog has a diagram for each part: the anatomy of a signed request, polling against push, the retry ladder, duplicates, out-of-order delivery and secret rotation.

A customer pays for order ord_8412. The shop sends your backend a webhook, and your handler
awards loyalty points, emails a receipt and calls the warehouse — which takes eleven seconds. The
sender's timeout is ten, so it marks the delivery failed and sends it again. The customer gets two
receipts and double points, and your logs show two perfectly healthy 200s.

Nothing in that story is a bug in the usual sense. Every line of the handler did what it said. It
is what happens when code written for a function call meets a protocol built on retries — and it
is the first of six failures this guide walks through, after the basics every one of them depends
on.

What is a webhook?

A webhook is an HTTP POST that one system sends to a URL another system registered, at the
moment an event happens. The receiver registers once — "when an order is created, tell me at
this URL" — and from then on the sender calls it every time. This is what one looks like, signed
the way the open Standard Webhooks
specification describes:

POST /webhooks/orders HTTP/1.1
Host: api.example-shop.com
Content-Type: application/json
Content-Length: 186
webhook-id: msg_2mQ8hZk3Xv9aR1cT
webhook-timestamp: 1789983612
webhook-signature: v1,pX/4CnjtyRIdKhtP4FEBLl0a8pgk0IErOOHRkgsduog=

{
  "type": "order.created",
  "timestamp": "2026-09-21T09:40:12Z",
  "data": {
    "order_id": "ord_8412",
    "customer_id": "cus_2291",
    "total": 42.50,
    "currency": "EUR"
  }
}

The signature is real. It was made with the secret
whsec_DioEXswemwdu+ScHzpEImNzhtrASzXe0vxCXPBmwuTM=: paste the body, that secret and the three
headers into the signature verifier and it will say the signature is
valid — and that the timestamp is far too old to accept. Both answers are correct, and the second
one is a security feature.

Webhook vs API polling

The alternative to being told is asking: calling the other system's API on a timer to see whether
anything changed.

Shorten a polling interval and you pay in requests and rate limits; lengthen it and you pay in
lag. A webhook costs one request per event. But read the last row twice: polling fails safe,
because a missed poll is repeated by the next one, while a webhook fails however the sender
decides. That is why polling keeps a job in a good webhook integration — as the slow
reconciliation that catches whatever the webhooks missed.

Sending webhooks: what the sender decides

Event types. Conventionally resource.verb in the past tense: order.created,
invoice.paid. The Standard Webhooks spec asks for a full-stop delimited type, with a
timestamp and a data object next to it. Treat types as a public API: renaming one breaks every
receiver that routes on it.

Payloads. A full snapshot of the resource saves the receiver a round trip; a thin
notification carrying only an id makes it fetch the current state, which is never stale.
Stripe offers both and calls them exactly that.

Subscriptions. Which endpoint wants which event types. Sending everything to everyone just
makes every receiver filter traffic it never asked for.

Signing. Anyone can POST to a public URL, so the receiver needs proof the request came from
you, unaltered. That proof is an HMAC — a hash of the message keyed with a secret only the two of
you hold. Standard Webhooks signs webhook-id, webhook-timestamp and the raw body joined by full
stops, with HMAC-SHA256, and sends the base64 result after a v1, prefix. The secret is 24 to 64
random bytes, base64-encoded, with a whsec_ prefix.

A replay window. The timestamp is signed so that a captured request cannot be sent again
tomorrow: the receiver rejects anything older than a few minutes. The spec leaves the tolerance to
you; Stripe's libraries default to five minutes, a sensible place to start.

Timeouts and status codes. The spec recommends a request timeout "somewhere between 15 and
30s". 2xx is success and anything else is retried — including redirects, which
Stripe also counts as failures. Two codes carry extra meaning:
410 Gone asks the sender to disable the endpoint, 429 Too Many Requests asks it to slow down
(MDN has the full list).

Receiving webhooks: four jobs, in order

Verify the signature on the raw bytes

Verification means recomputing the HMAC and comparing. What breaks it most often is what you
compute it over: it must be the exact bytes that arrived.

Look at the example body. It says "total": 42.50. Parse it and serialise it back and you get
{"type":"order.created",…,"total":42.5,…} — no whitespace, no trailing zero. Same data, different
bytes, a completely different HMAC. Many frameworks parse JSON before your handler runs, so the
body you are handed is already the wrong one. Stripe puts it bluntly: "Any manipulation to the raw
body of the request causes the verification to fail."

In Node with Express, ask for the raw bytes on that one route:

import crypto from 'node:crypto';
import express from 'express';

const secret = Buffer.from(process.env.WEBHOOK_SECRET.replace(/^whsec_/, ''), 'base64');
const TOLERANCE_SECONDS = 5 * 60;

// rawBody is a Buffer: the bytes that arrived, before any JSON parser saw them.
function verifyWebhook(rawBody, headers) {
  const id = headers['webhook-id'];
  const timestamp = headers['webhook-timestamp'];
  const signatures = headers['webhook-signature'];
  if (!id || !/^\d+$/.test(timestamp ?? '') || !signatures) return false;

  // The replay window: a captured request is useless five minutes later.
  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > TOLERANCE_SECONDS) return false;

  const expected = crypto
    .createHmac('sha256', secret)
    .update(`${id}.${timestamp}.`)
    .update(rawBody)
    .digest();

  // Space-separated: during a secret rotation there is one signature per secret.
  return signatures.split(' ').some((entry) => {
    const [version, value] = entry.split(',');
    if (version !== 'v1' || !value) return false;
    const received = Buffer.from(value, 'base64');
    return received.length === expected.length && crypto.timingSafeEqual(received, expected);
  });
}

const app = express();
app.post('/webhooks/orders', express.raw({ type: 'application/json' }), (req, res) => {
  if (!verifyWebhook(req.body, req.headers)) return res.sendStatus(400);
  // …record the id, enqueue, answer (below)
  res.sendStatus(200);
});

The same in Python, where Flask's request.get_data() returns the untouched bytes:

import base64
import hashlib
import hmac
import os
import time

SECRET = base64.b64decode(os.environ["WEBHOOK_SECRET"].removeprefix("whsec_"))
TOLERANCE_SECONDS = 5 * 60


def verify_webhook(raw_body: bytes, headers) -> bool:
    msg_id = headers.get("webhook-id")
    timestamp = headers.get("webhook-timestamp", "")
    signatures = headers.get("webhook-signature")
    if not (msg_id and signatures and timestamp.isascii() and timestamp.isdigit()):
        return False

    # The replay window: a captured request is useless five minutes later.
    if abs(time.time() - int(timestamp)) > TOLERANCE_SECONDS:
        return False

    signed = f"{msg_id}.{timestamp}.".encode() + raw_body
    expected = base64.b64encode(hmac.new(SECRET, signed, hashlib.sha256).digest())

    # Space-separated: during a secret rotation there is one signature per secret.
    for entry in signatures.split():
        version, _, value = entry.partition(",")
        if version == "v1" and hmac.compare_digest(value.encode(), expected):
            return True
    return False

Three details are deliberate. The comparison is constant-time (timingSafeEqual,
compare_digest), as the spec requires: a plain == returns sooner the earlier two strings
differ, and leaks the answer a byte at a time. The secret is base64-decoded after whsec_ is
stripped — using the string as-is is a classic reason a correct implementation never verifies
anything. And the header may carry several signatures, which matters the day you rotate. If you
would rather not own this code, the Standard Webhooks project publishes libraries that do exactly
this for most languages.

Answer in milliseconds, work later

The sender is holding a connection open and counting.
GitHub
gives you ten seconds, then "considers the delivery a failure". Stripe publishes no number; it
says to return 2xx "before any complex logic that could cause a timeout", and to process events
from an asynchronous queue.

So the handler does the minimum: verify, record the event id, enqueue, return 200. The email, the
warehouse call, the loyalty points — the eleven seconds from the opening — happen in a worker,
where nobody is waiting and a failure is yours to retry.

Deduplicate on the event id

A sender that retries will eventually deliver something twice. The spec says webhook-id
"remains the same no matter how many times a webhook that has failed is retried", and tells
receivers to use it as an idempotency key. Other providers have their own: X-GitHub-Delivery,
X-Shopify-Webhook-Id, the event's id at Stripe. The robust version is a unique constraint and
an insert that is allowed to do nothing:

def accept(conn, msg_id: str, raw_body: bytes) -> bool:
    """Stores the event once. Returns False if this id was already accepted."""
    cursor = conn.execute(
        "INSERT INTO received_webhooks (msg_id, body) VALUES (%s, %s) "
        "ON CONFLICT (msg_id) DO NOTHING",
        (msg_id, raw_body),
    )
    return cursor.rowcount == 1

A duplicate still gets a 200. You have it already, and an error would only make the sender try
again.

Never assume order

Unless a sender explicitly promises ordering, assume there is none. Stripe says outright that it
"doesn't guarantee the delivery of events in the order that they're generated". Failure four below
is what to do about it.

Six ways webhooks break in production

1. Your endpoint is down: who retries, and for how long?

A deploy, a crashed pod, an expired certificate. The sender gets a connection error or a 5xx,
and what happens next is entirely its policy. Good senders retry with exponential back-off — each
wait longer than the last, with random jitter so a backlog of retries does not land in the same
second. The Standard Webhooks spec's example schedule starts immediately and runs for more than
three days.

Not every sender retries. GitHub "does not automatically redeliver failed webhook deliveries"
(GitHub);
you redeliver by hand or through its API. Our
comparison of Stripe, GitHub and Shopify
puts three real policies side by side.

Takeaway: know each sender's retry schedule before the outage, and never return 2xx for an
event you have not stored — a 200 is a promise that you have it.

2. Your endpoint is slow: a timeout is a retry you asked for

A timeout looks like downtime to the sender, with one nasty difference: your handler may have
finished the work. That is the opening story. A slow endpoint converts directly into duplicate
processing, and it gets slower under load — exactly when a burst of retries arrives.

Takeaway: acknowledge inside the budget, whatever the work costs. If you do not know your
handler's 99th-percentile latency, measure it before anything else.

3. The same event arrives twice

Duplicates come from retries after a lost response, from redeliveries someone triggered by hand,
and from the sender itself. Stripe's docs say so in one sentence: "Webhook endpoints might
occasionally receive the same event more than once."

Takeaway: check the id before any side effect, in the same transaction that records the event.
A check in memory, or one after the email has gone, is not a check.

4. Events arrive out of order

An order.created whose first attempt failed can arrive after the order.updated and
order.cancelled that followed it. Apply them in arrival order and a cancelled order comes back
to life.

Compare a version or an updated_at on the resource and ignore anything older than what you
hold — or treat the webhook as a nudge and fetch the current state from the sender's API. Be wary
of the event timestamp as a tiebreaker: Stripe notes its events record created in seconds, so
distinct events can share one.

Takeaway: every handler must produce the right state whatever order its events arrive in.

5. Signatures suddenly stop verifying

This arrives as a wall of 400s, and the cause is nearly always one of five:

• The body was parsed and re-serialised before verification.
• The wrong secret: test instead of live, or another endpoint's.
• A whsec_ secret used as a literal string instead of decoded.
• A drifting server clock, so every timestamp looks stale.
• A secret rotated with no overlap.

The last one is an outage you schedule yourself. If the old secret dies the instant a new one is
issued, every request between that moment and your deploy fails. Senders avoid it by signing with
both secrets for a while: the spec describes exactly this, and Stripe keeps the previous secret
valid for up to 24 hours, with one signature per active secret.

Takeaway: accept any valid v1 in the header, deploy the new secret inside the overlap, and log
why verification failed — stale timestamp, no matching signature, missing header — never a bare
400.

6. Events stop, and nobody notices

The quietest failure is the worst. The sender exhausts its retries and gives up. A subscription is
disabled after repeated failures — Shopify removes it
(Shopify). Or your own
handler catches an exception, logs it at debug and returns 200. From then on nothing errors;
events just stop, and the first symptom is a customer asking where their order went.

The defence is a delivery log on whichever side you control:

Takeaway: alert on the failure rate and on silence. An endpoint that received a thousand
events an hour yesterday and none today is not healthy just because nothing is failing.

Testing webhooks locally

See what arrives. A request bin is a throwaway URL that records every request sent to it. Our
free webhook tester is one, keeping the latest hundred requests for 24 hours: point a
provider at it and read the real headers and body before you write a handler.

Reach your laptop. A tunnel gives localhost a public HTTPS URL. Stripe's docs suggest ngrok,
and its CLI forwards events with stripe listen --forward-to localhost:4242/webhook.

Replay real payloads. A captured body and its headers are a test fixture that is exactly what
production sends:

curl -X POST http://localhost:3000/webhooks/orders \
  -H 'Content-Type: application/json' \
  -H 'webhook-id: msg_2mQ8hZk3Xv9aR1cT' \
  -H "webhook-timestamp: $(date +%s)" \
  -H "webhook-signature: v1,$SIGNATURE" \
  --data-binary @order-created.json

--data-binary sends the file byte for byte; plain -d strips newlines and changes the
signature. A replay needs a fresh timestamp and so a fresh signature — the replay window doing its
job. When a signature will not verify, the signature verifier checks
body, secret and header in your browser without sending them anywhere.

Break it on purpose. Return 500 and watch the retries arrive. Sleep past the timeout. Send
the same payload twice. Send order.updated before order.created.

Build it yourself, or put a gateway in front?

For one sender and one receiver, build it: verification, a table of seen ids and a queue are a
day's work, and the code above is most of it.

It gets harder as the edges multiply. Sending webhooks to your own customers means storing every
event before the first attempt, a retry scheduler that survives restarts, isolation so one slow
customer does not delay the rest, a delivery log they can read, replay and secret rotation.
Receiving from several providers means several signing schemes and retry policies. None of it is
exotic, but it is a system. A gateway is that system already built — at the price of one more
component that must stay up and that holds your signing secrets.

How Railhook handles it

Railhook is an open-source webhook gateway, MIT-licensed; these are its defaults. Outgoing
webhooks are stored before the first attempt and attempted up to seven times, waiting one minute,
five minutes, fifteen minutes, one hour, six hours and twenty-four hours — about thirty-one hours
in all, each wait jittered between half and one and a half times its tier. Webhooks it relays
onward for you get a shorter ladder of five attempts. By default every delivery carries the
Standard Webhooks headers shown above, and after a secret rotation both secrets sign for a 24-hour
grace window. Every attempt is logged with its status code and response, and any stored event can
be replayed. The docs have the details.

Provider behaviour quoted above was read from each provider's own documentation on the date at the
top of this article. These policies change; follow the links before you build on a number.

Originally published on the Railhook blog. Railhook is an open-source webhook gateway — send and receive, MIT, self-host with one command: github.com/vadymkykalo/railhook.