Kafka Beyond Basics

Consumer groups

Multiple consumers can share the work of reading a topic by joining the same group ID:

Topic (6 partitions)
├── P0 ─┐
├── P1 ─┼── Consumer A  (group=analytics)
├── P2 ─┘
├── P3 ─┐
├── P4 ─┼── Consumer B  (group=analytics)
└── P5 ─┘

• Kafka assigns each partition to exactly one consumer in the group.
• Add a third consumer → Kafka rebalances so each gets 2 partitions.
• Two different groups reading the same topic each get all messages independently.

Replication

Every partition has a leader and 1+ followers on other brokers. Producers write to the leader; followers copy the data. If the leader crashes, a follower takes over — no data loss.

• Replication factor — usually 3 in production. RF=1 = no safety net.

Message ordering

Kafka guarantees order within a partition, not across partitions. To keep related messages in order (e.g. all events for the same order ID), set the key — Kafka hashes the key to always route to the same partition.

producer.send("orders", key=order_id, value=payload)

Real-world tip

Design your partition key carefully. A key with low cardinality (e.g. only 3 values) means most partitions sit idle. A key with billions of unique values spreads evenly — good.