I still remember the first time this happened to me in production.
A customer order was created. The database said so. The event was supposed to reach Kafka so downstream services could react. Everything looked fine on the dashboard.
Two hours later, a support ticket: the order existed, but nothing had happened downstream. No provisioning. No notification. Nothing.
The database was right. Kafka had no idea the order ever existed.
That day, I learned about the dual-write problem the hard way.
The scenario that breaks
Here's the code everyone writes at first. It looks innocent:
@Transactional
public void createOrder(Order order) {
orderRepository.save(order);
kafkaTemplate.send("orders", order.toEvent());
}
The intention is clear: save the order, then tell the world about it.
The problem is that these two operations live in two different systems. The database has its own transaction. Kafka has its own delivery guarantees. Nothing binds them together.
So here's what can happen:
- The database commit succeeds. Kafka is temporarily unavailable. The event is lost. Your database says the order exists. Your downstream services never learn about it.
- Kafka accepts the message. The database transaction then fails on commit (constraint violation, connection drop, deadlock). The event is out. The order is not. You've announced something that never happened.
Both scenarios are silent. No exception reaches your logs in a way that tells you what really went wrong. You only find out when someone downstream asks why they're missing data.
The fixes that don't fix anything
The instinct is to wrap both operations in a try-catch. Or to publish after commit with TransactionSynchronizationManager. Or to use Kafka transactions.
Each of these helps in a narrow case. None of them solves the fundamental problem:
You cannot atomically write to your database and to Kafka. They are two separate systems.
Any solution that pretends otherwise is going to leak under load, during network partitions, or when a broker restarts at the wrong moment.
So the real question is not "how do I make them atomic." It's:
How do I make my system recover correctly when they inevitably disagree?
That's a different design problem. And it has a known answer.
Transactional Outbox
The idea is simple. You stop trying to talk to Kafka from inside your business logic. Instead, you write the event to your own database, in the same transaction as the business data. Then a separate process reads those events and publishes them.
Concretely:
Application
├── DB transaction
│ ├── business data (order, subscriber, provisioning request)
│ └── outbox event (what should be published)
│
▼
Outbox Relay (polling or CDC)
│
▼
Kafka
The state change and the intent to publish become atomic. If the transaction commits, the event is guaranteed to be in the outbox. If the transaction rolls back, the event disappears with it.
There is no window where the database says one thing and Kafka says another.
What the code actually looks like
Here's the pattern in a Spring Boot + JPA + Kafka setup. I'm using PostgreSQL, but this works the same on Oracle.
The outbox table:
CREATE TABLE outbox_event (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(64) NOT NULL,
aggregate_id VARCHAR(64) NOT NULL,
event_type VARCHAR(128) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT now(),
published_at TIMESTAMP,
trace_parent VARCHAR(64)
);
CREATE INDEX idx_outbox_unpublished
ON outbox_event (created_at)
WHERE published_at IS NULL;
Two details matter here. The partial index keeps the relay fast as the table grows. The trace_parent column is what lets you keep distributed tracing alive across the outbox boundary — more on that later.
Writing the business data and the event together:
@Transactional
public void createOrder(Order order) {
orderRepository.save(order);
OutboxEvent event = OutboxEvent.builder()
.id(UUID.randomUUID())
.aggregateType("Order")
.aggregateId(order.getId().toString())
.eventType("OrderCreated")
.payload(toJson(order))
.traceParent(currentTraceParent())
.build();
outboxRepository.save(event);
}
Notice what's not happening: no kafkaTemplate.send(). The method has no idea Kafka exists. That's the point.
The relay:
@Scheduled(fixedDelay = 500)
@Transactional
public void publishPendingEvents() {
List<OutboxEvent> batch = outboxRepository
.findTop100ByPublishedAtIsNullOrderByCreatedAtAsc();
for (OutboxEvent event : batch) {
kafkaTemplate.send("orders", event.getAggregateId(), event.getPayload())
.whenComplete((result, ex) -> {
if (ex == null) {
event.markPublished();
}
});
}
}
Kafka down? The events pile up in the outbox. Nobody loses anything.
Kafka comes back? The next poll drains the backlog.
The business transaction has no idea any of this happened.
The parts nobody warns you about
The pattern is not complicated. What gets people is what happens next.
Ordering. If events for the same aggregate must be processed in order, your relay has to publish them in the order they were written. But if you scale the relay horizontally, two workers can grab overlapping batches and reorder things. The fix is SELECT ... FOR UPDATE SKIP LOCKED or partitioning by aggregate ID.
Consumer idempotence. Kafka is at-least-once. The outbox guarantees your event will eventually be published — but "eventually" might mean twice if the relay crashes between the publish and the markPublished(). Your consumer has to handle that. A processed_events table with a unique constraint on event_id is the cheapest answer.
Distributed tracing. This one bit me. When the HTTP request that triggered the order ends, the trace ends with it. The outbox relay runs later, in a different thread, possibly on a different instance. If you don't carry the trace context through the outbox table, your traces stop at the HTTP boundary.
That's why trace_parent is in the schema. The relay reads it and starts a new span linked to the original trace.
Table growth. The outbox is not a log you keep forever. Purge published events on a schedule — a daily job that deletes rows where published_at < now() - 7 days is fine for most systems.
Breaking Kafka on purpose
The moment the pattern clicked for me was when I deliberately broke Kafka in a test environment and watched the system recover.
Kill the broker mid-transaction. Watch the outbox fill up. Restart Kafka. Watch the relay drain it. The business data is already in the database. The events catch up. Nobody has to intervene.
That's the real test of the pattern. Not "does it work when everything is healthy." Anyone can write code that works when everything is healthy.
The test is: does it converge back to a correct state after Kafka, the database, or the network misbehaves?
If the answer is yes, the architecture is doing its job.
Where this comes from
I work in telecom provisioning. Number portability, 5G core migration, BSS integration. In this world, a provisioning request travels through six or seven systems, some of which are owned by other companies. The network doesn't always answer. A timeout doesn't mean failure. A retry can duplicate work.
The outbox pattern is one of the tools that makes this survivable. It doesn't eliminate failure. It makes failure recoverable.
I've packaged the pattern, along with Sagas, idempotent consumers, SLA timers, and reconciliation, in a runnable example:
The Outbox implementation is in the repo, with Kafka configured to be breakable on purpose. The tests show what happens when you kill the broker mid-transaction.
If you're building anything that writes to a database and publishes to a broker, take twenty minutes to read the code. Then try to break it. That's where the learning happens.
Next in this series: Kafka is at-least-once. Your business logic should be idempotent. I'll cover processed_events, unique constraints, safe replays, and how to design a consumer that can survive a duplicate delivery without corrupting state.
