In this post: what building a reliable webhook delivery system in Go and Kafka, my first project with both, actually taught me. Skip to What Kafka taught me if you want the meatier part.
A few weeks ago I got tired of reading about Go and Kafka and decided to just build something with both. Not a todo app. Something with real failure modes baked into the requirements from day one.
So I built webhook-delivery: a service that accepts webhook events over HTTP and reliably delivers them to customer endpoints. Retries, exponential backoff, a dead-letter queue, per-endpoint ordering, idempotency, concurrency limits, per-host circuit breaking.
This post isn't the happy-path story. It's the moments where my mental model was flat out wrong and the system, or the benchmark numbers, had to correct me.
Why webhook delivery
On the surface it's "accept a POST, forward a POST." But say the word "reliably" and you inherit a whole distributed systems curriculum for free:
- What happens when the customer's endpoint is down?
- What happens when it's slow instead of down?
- What happens when two events for the same customer arrive out of order?
- What happens when your own process crashes mid-delivery?
- How do you even know if a delivery failed, versus the response just got lost?
None of these have a clean answer. That's exactly what I wanted to sit inside for a while.
The shape of the system
POST /events ─► API ─► events ─► delivery workers ─► POST to endpoint
│ ok → commit
└ fail → retries ─► retry worker ─► waits, redelivers
│ exhausted / bad data → dead-letter
The API validates the request, publishes to a Kafka topic (events), returns 202. Delivery workers consume that topic, group messages by an orderingKey so a customer's events stay in order, and POST them out. Failures go to a retries topic with exponential backoff. Permanent failures and events past a max age go straight to a dead-letter queue.
That paragraph took about three weeks to actually get right. Almost all the real learning happened in the gap between "this looks correct" and "this is correct."
What Go taught me
I'd read about goroutines and channels before. I hadn't had to reach for them under pressure, and that's a different kind of understanding.
Channels as semaphores. A chan struct{} with capacity N is a free, composable concurrency limiter. I needed two levels of it, a global cap on in-flight deliveries and a per-host cap so one flaky endpoint couldn't eat the whole pool, and both turned out to be the same primitive:
func acquire(ctx context.Context, sem chan struct{}) error {
select {
case sem <- struct{}{}:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
The part that didn't click until I wrote a test for it: drop the ctx.Done() branch and a goroutine can block forever on a saturated semaphore, even after the parent context is cancelled. Nothing wakes it up.
func TestDeliverGroupUnblocksFromSaturatedSemaphore(t *testing.T) {
// ... a full host semaphore, then cancel the context ...
select {
case err := <-done:
if err == nil {
t.Fatal("deliverGroup returned nil for an undelivered group")
}
case <-time.After(2 * time.Second):
t.Fatal("deliverGroup is stuck acquiring a saturated semaphore after cancellation")
}
}
Takeaway: if a blocking channel op doesn't have a ctx.Done() escape hatch, it's not a semaphore, it's a deadlock waiting for a bad day.
The rest of the Go side was less about discovering new concepts and more about discovering how much they matter under real conditions. A few worth naming quickly:
-
Nested locking needs
deferdiscipline. The delivery path has a per-host semaphore, a global semaphore, and a mutex around breaker state, all nested. Writing that as a flat sequence ofLock/Unlockwith early returns is how you leak a lock. Wrapping each critical section in its own function literal givesdefera clean scope, and I use this pattern everywhere now:
if err := func() error {
mu.Lock()
hostChan, ok := perHostSem[host]
// ...
mu.Unlock()
if err := acquire(ctx, hostChan); err != nil {
return err
}
defer func() { <-hostChan }()
// ...
return nil
}(); err != nil {
return err
}
Interfaces make Kafka disappear from your test suite.
Workerdepends on a one-methodPublisherinterface, not*kafka.Producer. Every test uses arecordingPublisherthat just appends to a slice. That's the entire reason 45 tests run in about 3 seconds with-raceon and no Docker container in sight.errors.Asover string matching. Classifying a failure as retryable or permanent needed to see through wrapped errors:
func IsPermanent(err error) bool {
var statusErr *StatusError
if !errors.As(err, &statusErr) {
return false
}
if statusErr.StatusCode < 400 || statusErr.StatusCode >= 500 {
return false
}
return statusErr.StatusCode != http.StatusRequestTimeout &&
statusErr.StatusCode != http.StatusTooManyRequests
}
Small thing, but it's the difference between code that breaks the moment someone wraps the error with fmt.Errorf("...: %w", err) and code that doesn't care how many layers of wrapping happened.
What Kafka actually taught me
This is where most of my wrong assumptions lived. Kafka's mental model is genuinely different from a queue, and I hit that difference in ways that only showed up under load.
An offset commit is a watermark, not a checklist
The single biggest "oh" moment of the project. I assumed committing an offset meant "these specific messages are done," like checking items off a list. It doesn't. It means "everything up to and including this position is done."
That distinction seems pedantic until you try committing per-key-group as each group finishes, which I did, on a bad first draft. Here's the failure mode: key A's group finishes fast, key B's group is sitting right behind it in the same batch and still retrying. Commit A's high offset and you've silently marked B's still-in-flight messages as done too, because the offset is a single number, not a set.
My README has a note to self about this, because I don't trust myself to remember it on the next project:
The obvious fix for the batch barrier, committing per group as it finishes, is wrong. A Kafka offset commit is a watermark, not a set, so committing a fast group's highest offset marks a slower group's earlier messages as done while they are still in flight.
The actual design commits a whole batch only once every message in it reaches a terminal state. That has a real cost, the batch runs at the speed of its slowest destination, but at least it's not lying about what's durable.
Takeaway: if you're building anything on Kafka that needs partial-batch semantics, assume the naive per-item commit is wrong until proven otherwise.
Partition keys give you ordering, but only per key, and it's brittle
Kafka guarantees order within a partition, and hashing the same key to the same partition consistently is what makes "order per customer" work as "use the customer as the partition key." Sounds straightforward. What I underestimated was how much care that guarantee then demands downstream:
- If one message in a customer's group fails, every message behind it in that batch has to be deferred too, or a later event delivers before an earlier one that's still retrying.
- The consumer batch fetch has to preserve order within a group when replaying it.
- None of this is enforced by Kafka. It's enforced by discipline in your own worker loop.
if received != 1 {
t.Fatalf("endpoint received %d requests, want 1; messages behind the failure were delivered out of order", received)
}
That assertion message is a note to my future self about a bug I actually shipped once.
Batching is invisible until it's the whole bottleneck
I loaded the API at increasing concurrency and got a suspiciously perfect linear relationship: 20 concurrent submitters, 20 events/sec accepted. 60 submitters, 60/sec. That's not what a resource limit looks like, real capacity limits are noisy and plateau. A perfectly linear number is the signature of a fixed per-request delay.
The cause: kafka-go's writer flushes a batch on BatchSize or BatchTimeout, whichever comes first, and BatchTimeout defaults to one second. My API handler publishes exactly one message and blocks on the ack. The batch never filled by size, so every publish sat there for the full second before Kafka even saw it.
accept rate, concurrency 20: 20/s → 1,350/s
accept rate, concurrency 60: 60/s → 4,043/s
first-attempt p50: 1,085ms → 16ms
One config line (PRODUCER_BATCH_TIMEOUT=10ms) fixed a 67x throughput ceiling. RequiredAcks: RequireAll and a synchronous write path meant durability was never in question, so this was purely a batching knob I didn't know existed. I would not have found it by reading the code. The code looked fine. It only showed up because the numbers were suspiciously round.
Consumer groups don't coordinate with each other
events and retries are drained by two independent consumer groups. That's the right design, but it means there's no ordering guarantee across the boundary between "things failing" and "things recovering." When a circuit breaker's cooldown ends and a bad host comes back healthy, fresh events sail straight through events while much older events are still waiting out their backoff in retries. The backoff ladder schedules the oldest deferred messages the furthest out, and the outage guarantees a backlog exists at exactly the moment direct delivery resumes.
So the inversion isn't a bug to fix. It's a structural property of two independently scheduled paths. I measured it instead of asserting it away:
chaos inversions: 8 (range 0-34) across 400 keys, when hosts flap
A test demanding zero here would be testing the wrong thing.
The connection pool is not sized for you
http.DefaultTransport defaults to 2 idle connections per host. My delivery worker was configured for a concurrency of 5 per host. Every request past the second one on a given host tore down its TCP connection instead of reusing it: handshake, discard, repeat. Sizing MaxIdleConnsPerHost to match the configured concurrency cut discarded connections from 741 to 455 per 3,000 deliveries.
The interesting part: on loopback, this produced zero latency improvement. A local handshake is essentially free, so the fix was invisible in my own numbers even though it was unambiguously correct. I almost cut it from the README because "no measurable delta" felt like a null result. I kept it in, because implying every optimization should show up as a number on my machine is its own kind of dishonesty. The saving only exists once there's real network latency in the loop, which docker-compose on a laptop doesn't give you.
Takeaway: a correct fix with no visible benefit in your test environment isn't a wasted fix. It's a fix that's waiting for a more realistic environment to matter.
The circuit breaker: small state machine, subtle interactions
The breaker itself is genuinely small, three states, a failure counter, a cooldown:
func (b *Breaker) Allow() bool {
switch b.state {
case Open:
if b.now().Sub(b.openedAt) >= b.cooldown {
b.state = HalfOpen
return true
}
return false
case HalfOpen:
return false
default:
return true
}
}
What made it interesting wasn't the state machine. It was deciding what doesn't count as a failure against it. An event past its max age gets dead-lettered, but that's my decision to give up, not the host's fault, so it must never trip the breaker or cost a retry. A permanent 4xx rejection shouldn't count against a host's health either, the payload is wrong, not the destination.
Getting this wrong the first time meant a single customer sending malformed payloads could open a breaker and take down delivery to an otherwise healthy endpoint. Obvious once you say it out loud. Invisible while you're writing the happy path.
Load testing found bugs that reading the code never would
I built a side harness for this: a fault-injectable receiver (fixed status, hang for N seconds, flap between failing and succeeding on a cycle) plus a load generator with burst and steady profiles. Both real bugs above, the batch timeout and the connection pool, only showed up under sustained load, and only because I was looking at the shape of the numbers, not just whether they were fast enough. A suspiciously linear accept rate and a suspiciously round connection count were the tells. Reading the worker loop in isolation, everything looked correct. And it was, locally, at low concurrency. Which is exactly the trap.
If I could tell past-me one thing starting this project: write the load generator early, and look for numbers that are too clean, not just numbers that are too slow.
What I'd tell someone starting their first Go + Kafka project
- Build the failure paths first. Retry, DLQ, and breaker logic is where almost all the real design decisions live. The happy path is a rounding error.
- Table-driven tests pay for themselves immediately once you have more than two branches of behavior to verify, and in Go they're nearly free to write.
-
Every blocking channel op needs an escape hatch. No
ctx.Done()case means no semaphore, just a deadlock waiting for a bad day. - Kafka's guarantees are narrower than they sound. Ordering is per partition, not global. A commit is a watermark, not a checklist. Two consumer groups don't know about each other. None of this is a flaw in Kafka, it's just not what "queue" implies if you're coming from somewhere else.
- Benchmark before you trust your own reasoning. I was confident the code was correct in both bug cases above. It compiled, it passed tests, it looked right. It took real load and a suspiciously clean number to prove otherwise.
The repo, including the benchmark harness, raw JSON results, and all the tests, is here: github.com/Dev-Bilaspure/webhook-delivery. If "reliable" is part of your spec, that requirement alone will teach you more than the framework docs ever will.
