"At-least-once delivery" is four words that hide a dozen production incidents. So are "consumer lag", "dead letter queue", and "exponential backoff". Everyone uses the vocabulary; the engineers who are calm during the incident are the ones who have watched the mechanisms run, not just read the definitions.
Watching them run is cheaper than it sounds. Below are three free browser simulators, each covering one leg of asynchronous delivery: messages between services, webhooks to the outside world, and the rate limits everything eventually hits. For each, the specific failure worth causing on purpose. Disclosure: I help build these; all three are free, no signup.
Leg 1: the queue between your services
The Message Queue Simulator models both of the shapes you will meet, a Kafka-style topic with partitions and a RabbitMQ-style exchange with queues, and runs six scenarios against them. The order that builds the intuition:
- Normal operation first, to see the baseline: producers in, consumers out, depth stable.
- Consumer lag: production outpaces consumption and you watch the gap grow. Lag is the queue metric on-call actually pages on, and seeing it as a growing backlog (rather than a number on a dashboard) explains why "lag is growing" is an incident and "lag is high but stable" might be lunch-first.
- Backpressure: the scenario floods the queue faster than consumers drain it and you watch the backlog pile up. The simulator shows the overload accumulating; the production lesson it points at is that something has to give, and slowing producers deliberately beats buffering until something breaks.
- Dead letter handling: a poison message gets parked in a DLQ instead of blocking the stream (the simulator routes it there directly; real systems usually retry a few times first). It answers the question every team eventually argues about: what happens to the message that cannot be processed? It goes somewhere a human will look, and the stream keeps moving.
- Ordering guarantees: the scenario that corrects the most common Kafka misconception. Order holds within a partition, not across the topic, and the scenario makes you see why the topic-wide order you assumed does not exist.
- Rebalancing: consumers join or leave and you watch partition ownership get reassigned. Historically that meant a stop-the-world pause (modern Kafka rebalances incrementally, so the blip is smaller than its reputation), and this scenario is why deploying a consumer group shows up on the lag graph at all.
Six scenarios, maybe twenty minutes, and the difference between Kafka's log model and Rabbit's exchange model stops being trivia: it becomes "which failure modes do I inherit".
Leg 2: the webhook to someone else's server
Queues connect services you control. Webhooks deliver events to servers you do not, which is a harder trust and reliability problem.
The Webhook Delivery Simulator puts you on both sides of the delivery. You control how the receiving endpoint behaves: return 200, 400, 429, 500, time out, or fail intermittently, and watch what a well-built sender does about each. The simulator's sender follows an illustrative policy: a 500 or timeout gets retried with backoff, a 400 does not (the request is wrong, resending will not fix it), and a 429 gets slowed down. Real senders vary (Stripe and Svix retry any non-2xx, for example), which is itself the lesson: your endpoint's status code is not a log line, it is an instruction to the sender, so know your provider's mapping.
The second half is the security exercise: HMAC signatures. The simulator signs each delivery with a shared secret, and then lets you tamper: edit the body after signing and verification fails; replay a stale delivery and the timestamp check catches it. Doing the tampering yourself makes the point better than a paragraph about it: an unverified webhook endpoint is an open API that trusts whoever knocks.
If you run webhook consumers in production, the exercise translates directly: verify the signature, durably record the event, return 2xx, then do the expensive work asynchronously; treat 4xx as your bug; expect duplicates because retries exist. (We wrote a longer piece on the sending side, what it takes to deliver a webhook in production, if you want the full depth.)
Leg 3: the rate limit everything hits
Eventually your service is the client, calling an API that pushes back. The Rate Limit Simulator puts you on the client side of a 429: you fire requests at a limited API and choose your strategy: no backoff, fixed delay, or exponential backoff, while the counters track successful versus throttled requests.
Run "no backoff" first to see the naive baseline: throttled requests simply fail and burn budget. Then compare fixed delay against exponential backoff on the same traffic and watch the throttled count differ. The response also carries a Retry-After header, worth knowing about even though it is optional in the wild: when an API does send it, that is the server telling you when to come back, and production clients should prefer it over locally-invented delays. The lesson compounds with the webhook leg: a 429 is not an error to log, it is scheduling information.
The through-line
All three legs are the same idea wearing different clothes: delivery is a negotiation, not a fire-and-forget. The queue negotiates with lag and backpressure, the webhook sender negotiates with status codes and signatures, the API client negotiates with 429s and backoff. Systems fail when one side pretends the negotiation is not happening: the producer that ignores lag, the receiver that acknowledges events it never durably recorded, the client that retries without waiting.
A session across the three simulators gives you the vocabulary as experiences instead of definitions. They are part of 50+ free DevOps games and simulators. To go deeper, the Kafka design docs and Stripe's webhook documentation are the two references that repay reading after you have the intuition.
