How do you troubleshoot consumer lag in a Kafka application?

Consumer lag — the gap between the latest offset produced to a partition and the offset a consumer group has actually processed — is the standard signal that a consumer group isn't keeping up, and the troubleshooting path starts with measuring it precisely before guessing at a cause.

kafka-consumer-groups.sh --describe --group order-processors --bootstrap-server localhost:9092

1. Check if lag is growing or just non-zero — some lag is normal under bursty load; steadily increasing lag over time is the real signal of a problem.
2. Check per-partition lag, not just the group total — lag concentrated on one or two partitions usually points to a hot key or uneven partition assignment, not a group-wide capacity problem.
3. Check processing time per record — slow downstream calls (a database write, an external API) inside the consumer loop are a very common root cause.
4. Check consumer count versus partition count — if the group already has as many consumers as partitions and is still falling behind, only adding more partitions and consumers together will actually add parallelism.
5. Check for frequent rebalances — a group that keeps rebalancing spends time reassigning partitions instead of processing, which itself can look like lag.