Explain the Kafka message flow end-to-end

Question

Walk me through what happens when a Kafka producer sends a message and a consumer receives it.

Answer

1) The **producer** serializes the message and computes a partition — usually `hash(key) % partitions` (or round-robin if no key). 2) It sends the record to the **broker that owns the leader replica** of that partition; the broker appends the record to its local log, replicates it to follower brokers, and (based on `acks`) returns an offset. 3) The record now sits at that offset in the partition, immutable. 4) A **consumer** in a consumer group is assigned that partition by the group coordinator, tracks its position via a **committed offset** (stored in the internal `__consumer_offsets` topic), and calls `poll()`. 5) Kafka returns a batch of records starting at that offset; the consumer processes them, then commits the new offset — either automatically or explicitly. If the consumer crashes before commit, another consumer in the group picks up from the last committed offset and reprocesses.