Archprimer
ConceptIntermediate 7 min read

Kafka

A distributed, append-only commit log that streams high-throughput events across partitions with durable, replayable storage and consumer-group scaling.

streamingevent-drivenlogmessagingthroughput

By Ratish Jain · Updated

Diagram with 4 components: Producer, Kafka Broker, Group A, Group B. Connections: Producer → Kafka Broker (1 · publish); Kafka Broker → Group A (2 · poll); Kafka Broker → Group B (3 · poll).
ServiceQueue
Producers append to partitioned topics; consumer groups read independently.

Definition

Kafka is a distributed, append-only commit log for streaming events. Records are written to the end of a topic, which is split into ordered partitions. Each record gets a monotonically increasing offset, and data is retained on disk for a configured window so it can be re-read.

Why It Exists

Traditional brokers delete a message once it is consumed, so replaying history or adding a new consumer after the fact is hard. Kafka decouples producers from consumers by keeping the log itself as the source of truth. This lets many independent teams read the same stream at their own pace, and lets systems reprocess old data after a bug fix or a new feature.

How It Works

Producers pick a partition (by key hash or round-robin) and append records. Each partition is replicated across brokers; one replica is the leader and the rest are followers that stay in sync (the ISR set). Consumers belong to a consumer group: partitions are divided among group members so each partition is read by exactly one member, giving parallelism. Consumers track their own offset, so reading is just a sequential disk scan — cheap and fast. Ordering is guaranteed only within a partition, not across the topic.

What It's Used For

  • Event streaming and activity logs (clicks, page views, IoT telemetry)
  • Decoupling microservices via durable pub/sub
  • Log aggregation and metrics pipelines feeding data lakes
  • Change-data-capture and event sourcing
  • Buffering spikes before slower downstream sinks

Partitions & Offsets

Diagram with 6 components: Producer, Partition 0, Partition 1, Partition 2, Group A, Group B. Connections: Producer → Partition 1 (1 · append); Producer → Partition 0 (append); Producer → Partition 2 (append); Partition 1 → Group A (offset 5); Partition 1 → Group B (offset 2).
ClientQueueService
Producers append to partitions; each consumer group tracks its own offset.

Tradeoffs

Pros
  • Very high throughput via sequential disk I/O and zero-copy
  • Durable, replayable log — new consumers can rewind
  • Horizontal scale through partitions and consumer groups
  • Strong ordering within a partition
Cons
  • No per-message routing logic; consumers filter themselves
  • Operationally heavy (brokers, replication, rebalancing)
  • Ordering across partitions is not guaranteed
  • Overkill for simple task queues or low-volume workloads

When to Use / Avoid

Use when
You need high-throughput streaming, replay, or multiple independent consumers of the same event stream.
Avoid when
You need complex routing, per-message TTL/priority, or a lightweight job queue — reach for RabbitMQ.
Real-world
LinkedIn (its origin), Uber, and Netflix use Kafka as the backbone for trillions of daily events.

Interview Questions

How does Kafka guarantee ordering?

Only within a single partition. Records with the same key hash to the same partition, so per-key ordering holds; there is no global ordering across a topic's partitions.

What happens when a consumer group adds a member?

A rebalance redistributes partitions across members so each partition still maps to exactly one consumer. Excess consumers beyond the partition count sit idle.

60-Second Revision
  • →Append-only log split into ordered partitions with offsets.
  • →Retention lets consumers replay; broker doesn't delete on read.
  • →Consumer groups give parallelism — one partition per member.
  • →Ordering is per-partition only, keyed by record key.
  • →Best for high-throughput streaming, not fine-grained routing.

Related

Everything on Archprimer is connected — follow the graph.