Kafka: Consumer lag keeps growing

Step-by-step Solution

1. 1Confirm the lag is real by running the command twice, 30s apart. If lag is rising, consumers can't keep up with producers.
2. 2Check how many consumers are in the group and how many partitions the topic has. A group with 2 consumers can never read more than 2 partitions in parallel — even if the topic has 20. Scale consumers up until you hit the partition count.
3. 3Profile the consumer: is it CPU-bound (heavy processing per record) or I/O-bound (slow DB writes)? For I/O-bound, batch downstream writes; for CPU-bound, add consumers.
4. 4Increase `max.poll.records` and `fetch.max.bytes` — bigger batches per poll cut per-message overhead significantly.
5. 5If a single partition is far ahead of others, the key distribution is skewed. Re-key the producer to a higher-cardinality key or add partitions (but be careful — re-partitioning breaks strict per-key ordering across the transition).
6. 6As a **last resort** during an incident, reset the consumer group offset closer to `latest` (`--reset-offsets --to-latest`) to catch up quickly, and reprocess the skipped messages from a snapshot later.