Apache Kafka: Understanding Event Streaming Architecture

Apache Kafka has become the de facto standard for building real-time data pipelines and event-driven applications. Originally developed at LinkedIn and later open-sourced under the Apache Software Foundation, Kafka is a distributed event streaming platform capable of handling trillions of events per day. This post explores the core concepts, architecture, and practical patterns for working with Kafka.

What Is Event Streaming?

Event streaming is the practice of capturing data in real time from event sources—such as databases, sensors, mobile devices, and applications—in the form of streams of events. These streams are stored durably for later retrieval, processed and reacted to in real time or retrospectively, and routed to different destination technologies as needed.

Unlike traditional messaging systems that treat messages as transient, Kafka treats events as a durable, replayable log. This fundamental design choice unlocks powerful architectural patterns.

Core Concepts

Topics and Partitions

A topic is a category or feed name to which events are published. Topics are split into partitions, which are the unit of parallelism and scalability in Kafka.

• Each partition is an ordered, immutable sequence of records.
• Records within a partition are assigned a sequential ID called an offset.
• Ordering is guaranteed only within a partition, not across partitions.

Topic: user-events
├── Partition 0: [msg0][msg1][msg2][msg3]...
├── Partition 1: [msg0][msg1][msg2]...
└── Partition 2: [msg0][msg1][msg2][msg3][msg4]...

Producers and Consumers

Producers publish events to topics. They can choose which partition to write to, either explicitly, via a partition key (records with the same key always land in the same partition), or through round-robin distribution.

Consumers read events from topics. They are organized into consumer groups, where each partition is consumed by exactly one consumer within a group. This enables horizontal scaling of consumption.

Consumer Group A
├── Consumer 1 → Partition 0, Partition 1
└── Consumer 2 → Partition 2

Brokers and Clusters

A Kafka broker is a single server in a Kafka cluster. Brokers store data and serve client requests. A production cluster typically consists of multiple brokers to provide fault tolerance and scalability.

Replication and Fault Tolerance

Kafka achieves durability through partition replication. Each partition has one leader and multiple followers (replicas).

• All reads and writes go through the leader.
• Followers replicate the leader's log to stay in sync.
• The set of replicas caught up with the leader is called the In-Sync Replicas (ISR).

If a leader fails, one of the in-sync replicas is automatically promoted to leader, ensuring high availability.

The acks producer setting controls durability guarantees:

# Wait for all in-sync replicas to acknowledge (strongest guarantee)
acks=all

# Wait only for the leader
acks=1

# Fire and forget (highest throughput, lowest durability)
acks=0

A Simple Producer Example

Here is a basic Java producer that publishes events to a topic:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    for (int i = 0; i < 100; i++) {
        ProducerRecord<String, String> record =
            new ProducerRecord<>("user-events", "user-" + i, "login-event");
        producer.send(record, (metadata, exception) -> {
            if (exception != null) {
                exception.printStackTrace();
            } else {
                System.out.printf("Sent to partition %d at offset %d%n",
                    metadata.partition(), metadata.offset());
            }
        });
    }
}

A Simple Consumer Example

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "analytics-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("enable.auto.commit", "false");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(Collections.singletonList("user-events"));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
        for (ConsumerRecord<String, String> record : records) {
            System.out.printf("key=%s value=%s offset=%d%n",
                record.key(), record.value(), record.offset());
        }
        consumer.commitSync();
    }
}

Committing offsets manually (enable.auto.commit=false) gives you precise control over delivery semantics.

Delivery Semantics

Kafka supports three delivery guarantees:

Exactly