What is Kafka?

Kafka is a distributed log — imagine a giant, ever-growing notebook that many writers add to and many readers read from, in order.

The three actors

Producer  ---->  [ Topic:  orders ]  ---->  Consumer(s)

• Producer — writes messages to a topic.
• Topic — a named log (e.g. orders, payments, logs.error).
• Consumer — reads messages, at its own pace.

Partitions and offsets

A topic is split into partitions (usually 3-30). Each partition is an ordered, append-only log. Every message has an offset — a monotonically increasing number.

Topic: orders
├── Partition 0: [0][1][2][3][4] ...
├── Partition 1: [0][1][2][3]
└── Partition 2: [0][1][2][3][4][5]

Consumers remember their offset — they can rewind or fast-forward.

Real-world example

At an e-commerce site:

1. Checkout service publishes order-placed events.
2. Inventory service consumes them and decrements stock.
3. Email service consumes the same events and sends a receipt.
4. Analytics service consumes them for dashboards.

One producer, many independent consumers, all working from the same log.

When not to use Kafka

• Request/response APIs (use HTTP).
• Small volumes on a single machine (use RabbitMQ or SQS — simpler).
• You need queries and joins (use a database).