Consumer lag keeps climbing, consumers are flat out and brokers look fine. How do you find the bottleneck?

The 30-second answer

Lag grows when the arrival rate exceeds the group's processing capacity, which is roughly partitions divided by the time per record when each partition is processed sequentially. So I'd measure both. poll-idle-ratio near zero confirms the consumers are processing-bound, and per-stage timings show where the time goes, usually a per-record call to a slower downstream dependency. I'd also check the shape of the lag across partitions: uniform means systemic, a single partition means a hot key or a poison message. Mitigation is to stop scaling past the partition count, fix or cache the slow dependency, and skip stale low-value events if the business agrees. Then batch lookups and writes, process in parallel within each partition while keeping per-key order, and alert on lag in seconds and on capacity headroom.