Write the event into the same transaction as the row. Poll the table. Send to Kafka. Mark it sent. That's the whole pattern, and every version of it you'll find online stops right there, because at that level of description it's obviously correct — one commit, so a published fact can never disagree with the row that caused it.
I built one, ran it for real between two services, and hit three separate ways for "obviously correct" to still go wrong in practice. None of them are exotic. All three are the kind of thing that only shows up once something is actually polling a table and actually talking to a broker instead of living in a diagram.
Pitfall 1 — the poison message that never gets counted
The relay's job is: claim a batch of unsent rows, send each one, mark it SENT on a broker ack. The first version of that loop looked reasonable:
} catch (Exception e) {
log.warn("Outbox relay failed for message {}; row stays NEW and will be retried", messageId, e);
break; // keep ordering; remaining rows retried next pass
}
break instead of continue is deliberate and correct on its own — you want per-partition ordering preserved, so if row 3 fails, rows 4 through 20 shouldn't jump ahead of it. The bug is what's missing: nothing counts how many times row 3 has failed. If the reason it's failing is permanent — a broker that's actually down for that topic, a message the producer can't serialize, anything that isn't going to resolve itself — the relay retries the same row forever, on every poll, and every row behind it queues up behind a jam that will never clear on its own. No counter, no ceiling, no way to notice except staring at a growing NEW count in the table.
The fix is the boring one: give it a number to compare against.
int attempts = ((Number) row.get("attempts")).intValue() + 1;
if (attempts >= maxAttempts) {
jdbc.update("UPDATE outbox_message SET status='FAILED', attempts=? WHERE id=?", attempts, row.get("id"));
log.error("Outbox message {} parked as FAILED after {} attempts", messageId, attempts, e);
continue;
}
jdbc.update("UPDATE outbox_message SET attempts=? WHERE id=?", attempts, row.get("id"));
break;
After maxAttempts (default 10), the row is parked as FAILED and the loop moves past it — continue, not break, specifically for the row that's given up, so the jam it was causing clears while the ones behind it that might genuinely still succeed keep trying. A FAILED row is something a human can query, alert on, and retry by hand once the root cause is fixed. An infinitely-retried NEW row that never surfaces anywhere is not.
The part worth sitting with: the code that shipped without a counter passed every test that existed at the time. Tests exercised the happy path and one transient failure that resolved itself. Nothing exercised "this row will never succeed," because writing that test requires first believing a message can be permanently unsendable — which is exactly the assumption that's easy to skip when you're picturing the pattern as four sentences.
Pitfall 2 — the trace dies at the queue
HTTP calls carry a trace context automatically. order calls inventory's reservation endpoint, and whatever traced the request into order traces it straight through to inventory — that's what tracing instrumentation for HTTP clients is for.
A polled outbox message gets none of that. By the time the relay picks a row off the table, the HTTP request that originally caused the write is long finished, its context long gone. Kafka doesn't know what a trace is. Left alone, every event on the far side of a KafkaListener starts a trace of its own — inventory deducting a hold shows up in Jaeger as an orphan, with no link back to the checkout that paid for it.
The fix has to be manual, because there's no automatic HTTP-style propagation to lean on. The outbox row carries the trace id as a column, written at insert time from whatever's currently in MDC:
jdbc.update("INSERT INTO outbox_message (message_id, topic, message_key, payload, trace_id) VALUES (?,?,?,?,?)",
messageId, topic, key, jsonPayload, MDC.get("traceId"));
The relay reads that column back and attaches it as a Kafka header alongside the message id used for deduplication:
String traceId = (String) row.get("trace_id");
if (traceId != null) {
rec.headers().add(OutboxHeaders.TRACE_ID, traceId.getBytes(StandardCharsets.UTF_8));
}
And the consumer restores it before doing anything else, so every log line and span for the duration of processing that message is tagged with the trace that started at checkout:
var traceHeader = record.headers().lastHeader(OutboxHeaders.TRACE_ID);
String traceId = traceHeader == null ? null : new String(traceHeader.value(), StandardCharsets.UTF_8);
if (traceId != null) {
MDC.put("traceId", traceId);
}
try {
// ... process
} finally {
MDC.remove("traceId");
}
Three lines of column, three lines of header, three lines of MDC — and the payoff shows up as one line grepped out of two different services' logs:
traceId bd1a34b2a4665b240cf284c31db76708 crossed Kafka into the consumer log
Same id, order's side and inventory's side, proving the bridge actually held. Skip this and the trace doesn't error out or warn you — it just quietly stops, and "quietly stops" is the worst failure mode a debugging tool can have, because you don't find out until you're already looking for something and it isn't there.
Pitfall 3 — at-least-once means you will get it twice
Kafka's delivery guarantee is at-least-once, not exactly-once, and that isn't a rare edge case you might hit — it's the normal operating mode. A broker retry, a consumer rebalance, a producer resending after a slow ack: any of them redelivers a message the consumer already handled. The second copy of order-paid has to be a no-op. If it isn't, "convert the hold into a sale" runs twice and the stock ledger disagrees with reality.
The dedup row and the side effect it's guarding both have to commit in the same transaction, or the fix doesn't actually fix anything:
public boolean processOnce(UUID messageId, String consumerGroup, Runnable action) {
return Boolean.TRUE.equals(tx.execute(status -> {
int inserted = jdbc.update(
"INSERT INTO processed_message (message_id, consumer_group) VALUES (?,?) ON CONFLICT DO NOTHING",
messageId, consumerGroup);
if (inserted == 0) {
return false;
}
action.run(); // same transaction: the dedup row and the side effect commit together
return true;
}));
}
Splitting that into two transactions — insert the dedup row, commit, then run the side effect — reopens the exact gap this exists to close: a crash between the two either loses the effect (dedup row committed, side effect never ran) or blocks it forever (some other bug makes the side effect always fail, but the dedup row already says "done"). One transaction, both or neither.
processed_message is the fast path, and it's not the only layer here — the ledger's own unique constraint on (order_id, sku_id, type) is a second, independent check at the data level, so even a message that somehow slipped past deduplication can't double-insert a movement. Testing this for real means actually delivering the same message twice and watching nothing move the second time, not asserting it in the abstract:
// deliver the same messageId twice; Awaitility.during(2s) — not a snapshot —
// confirms the ledger row count and available/reserved stay flat across the whole window
A one-shot assertion right after the second delivery can pass by accident if the second message just hasn't been processed yet. Watching a window is the difference between "looks idempotent" and "is idempotent."
When you'd reach for Debezium instead
Everything above is the polling-outbox version: a scheduled query with FOR UPDATE SKIP LOCKED instead of a CDC connector tailing the write-ahead log. Debezium never misses a row and adds essentially no latency, and it costs you Kafka Connect — another JVM, another thing that can fall over, another gigabyte you don't get back. On a budget where the whole stack has to cold-start on a 6 GB box, that trade isn't close.
The honest trigger for switching: sustained throughput in the thousands of events per second, a latency requirement tighter than "within about a second," or an operations team that's already running Kafka Connect for something else, so the marginal cost is close to zero. None of those describe this system today. The schema doesn't care which way you go — outbox_message looks the same either way, which is the actual point of keeping the contract in the database instead of in whichever tool happens to be reading it.
This is part of a series on building a multi-vendor commerce platform. The transactional outbox and the idempotent-consumer helper both live in stallora-cloud-starter, Apache-2.0, and every pitfall above was found by actually running the thing against real Postgres and real Kafka, not by reading the pattern's description one more time.
