TL;DR: If you've spent years with Sidekiq, Kafka can look like "Sidekiq with extra steps". It isn't. Sidekiq moves jobs (commands to do something). Kafka stores events (facts about what happened) in a log that many readers can each read at their own pace. This is the concept map I wish I'd had when I moved from Sidekiq and SQS at work to building a Kafka system myself, including the Sidekiq features you'll have to rebuild yourself.
Where I'm coming from
My day job is mostly Ruby on Rails: background work runs through Sidekiq, and some integrations use SQS. Those tools are great, and for most Rails apps they're the right choice.
Then I built TxFlow, where a single payment event has to trigger five independent reactions (fraud, wallet, notification, audit, analytics). That's where my Sidekiq mental model started giving me wrong answers. Here's the map I ended up drawing.
The one-table version
| Sidekiq | Kafka | What actually changes |
|---|---|---|
Job (perform_async) |
Event / record | A job says "do X". An event says "X happened". |
| Queue | Topic + partitions | A queue is emptied as jobs are processed. A topic keeps records until retention expires. |
| Worker processes | Consumer group | Each group reads every event, independently. |
| Enqueue the same job N times for N handlers | N consumer groups on one topic | The producer doesn't know who's listening. |
| Built-in retries (25 over ~20 days) | You build it | Kafka has no per-message retry. |
| Dead set | Dead-letter topic | Also something you build. |
| No ordering guarantee | Ordering per partition | Key by user_id to keep one user's events in order. |
| Can't replay a finished job | Reset offsets and re-read | The log is still there. |
| Unique jobs (Enterprise / gems) | Idempotent consumers | Duplicates are normal, so you design for them. |
| Scale by adding threads/processes | Scale up to the partition count | 3 partitions = at most 3 active consumers per group. |
The rest of this post goes through the rows that surprised me.
1. Jobs vs events changes who owns the coupling
In Sidekiq, the code that creates a payment usually knows every follow-up:
# app/services/place_payment.rb
FraudCheckJob.perform_async(payment.id)
DebitWalletJob.perform_async(payment.id)
SendReceiptJob.perform_async(payment.id)
AuditLogJob.perform_async(payment.id)
Adding a fifth reaction means editing this file. With Kafka, the producer publishes one fact:
producer.produce("payments.initiated", key=user_id, value=json.dumps(event))
A new reaction is a new consumer group that subscribes to the topic. The producer never changes. That was the real reason I moved to Kafka for this project. Throughput had nothing to do with it.
2. A consumer group isn't a worker pool, it's a subscriber
This one confused me at first:
- Same group, many instances: the instances split the partitions between them (like Sidekiq processes sharing a queue).
- Different groups: each group gets a full copy of every event.
So "five things react to a payment" is five consumer groups, not five queues and not five job classes.
3. The retries you get for free in Sidekiq are gone
Sidekiq's error handling is so good it's easy to take for granted: 25 retries with exponential backoff over roughly 20 days, then the job goes to the Dead set (kept up to 10,000 jobs or 6 months), plus a sidekiq_retries_exhausted hook.
Kafka gives you none of that. If a consumer keeps failing on one record, it blocks that partition, because offsets are committed in order. In TxFlow I had to build:
- in-process retries with exponential backoff (3 attempts),
- a
payments.dlqtopic for records that still fail, - a DLQ handler service that stores failures in Postgres and shows them on a dashboard with a replay button.
It's not hard, but it's real work. Plan for it when you estimate a Kafka migration.
4. Delivery guarantees: different defaults than you'd think
A detail many Rails developers don't know: open-source Sidekiq fetches jobs with BRPOP, which removes the job from Redis when it's fetched. If the process is killed hard (kill -9, OOM) mid-job, that job is lost. On a normal shutdown, Sidekiq pushes unfinished jobs back. Sidekiq Pro's super_fetch closes that gap by keeping in-flight jobs in Redis until they finish.
With Kafka and manual offset commits, a crash mid-processing means the record is delivered again. You get at-least-once delivery by default, and the price is that duplicates are normal. Every consumer with side effects needs to be idempotent (I use a Redis dedup key plus a database primary key for the wallet debit).
5. Ordering is per partition, and you choose the key
Sidekiq makes no ordering promises between jobs. Kafka guarantees order within a partition, and the record's key decides the partition. TxFlow keys payment events by user_id, so all of one user's events land on one partition, in order. That's what you want for applying balance changes.
The flip side is that your parallelism is capped by partition count. You can't add a fourth active consumer to a group reading a 3-partition topic and expect it to do anything.
6. Replay is the superpower
A finished Sidekiq job is gone. A Kafka event stays in the log until retention expires. So you can:
- start a new consumer group months later and have it read history (as far back as retention allows),
- fix a bug in a consumer and reprocess a time range,
- rebuild a projection like analytics counters from scratch.
The test that made it click for me: stop one consumer, fire 10 payments, restart it, and watch it catch up while the other four groups never noticed.
When I'd stay on Sidekiq
Most of the time:
- the work is a command with one handler (send this email, resize this image),
- you want rich retries and a UI without building them,
- you don't need replay or multiple independent subscribers,
- your team doesn't want to run a broker.
Kafka earns its operational cost when you have one fact and many independent reactions, need replay, or need ordering per entity at scale. If that's not you, Sidekiq and Postgres will take you a very long way.
Question for Rails folks: what's the one Sidekiq feature you'd miss most if you moved to Kafka? My guess is the retry UI.
