What are the core concepts of Apache Kafka and how does it guarantee message durability and ordering?

How to think about it

Kafka is the de facto streaming backbone in modern data platforms. Its design separates producers, storage, and consumers — making it resilient and independently scalable.

Core concepts

Topic — a named, append-only stream of records. Producers write to topics; consumers read from them.

Partition — a topic is split into N partitions. Each partition is an ordered, immutable log stored on one or more brokers. Parallelism equals partition count.

Offset — each message within a partition has a monotonically increasing integer offset. Consumers track their own position via offsets, enabling replay and exactly-once semantics.

Consumer group — a set of consumers that collaboratively consume a topic. Each partition is assigned to exactly one consumer in the group. Adding consumers up to the partition count increases read parallelism.

Broker — a Kafka server that stores partition data. A cluster runs multiple brokers for fault tolerance.

# Spark Structured Streaming reading from Kafka
df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "broker:9092") \
    .option("subscribe", "user_events") \
    .option("startingOffsets", "earliest") \
    .load()

events = df.selectExpr("CAST(value AS STRING) as json_payload", "timestamp")

Durability: replication

Each partition has one leader and N-1 followers (replicas). Producers write to the leader; followers replicate asynchronously. A message is “committed” only after min.insync.replicas replicas acknowledge it.

Setting acks=all in the producer config means the broker only acknowledges after all in-sync replicas have written — strongest durability guarantee.

Ordering guarantee

Within a single partition, messages are strictly ordered by offset. Across partitions, there is no global order. To order by a key (e.g., user events for the same user), use a partition key — Kafka hashes the key deterministically to always route that key to the same partition.

# Producer: route all events for the same user_id to the same partition
producer.send("user_events", key=user_id.encode(), value=payload)

Retention and replay

Kafka retains messages for a configurable period (default 7 days) regardless of whether they have been consumed. Consumers can rewind their offset to replay events — a powerful feature for reprocessing pipelines after bug fixes.