A live-sports streaming platform pushes player telemetry (heartbeats, bitrate switches, rebuffer events) into Kafka: about 600k messages per second at the toss of a cricket final, on a 96-partition topic. The analytics consumer group feeding the live QoE dashboard is now two minutes behind. Producer error rate and latency are normal, yet the 64 consumer pods sit at 10–15% CPU.
The interviewer asks“Producers are healthy, the consumers are barely using CPU, and lag keeps growing. Walk me through how you work out what the consumers are waiting on, and what you change during the match versus after it.”
Lag with idle consumers means they are waiting, not working, so scaling them is usually wrong. I'd look at lag per partition first: if it sits on one or two partitions it's a hot key or a stuck member; if it's even, I check whether the group is rebalancing, whether threads are blocked on a downstream call or lock, and whether the brokers are throttling or reading from disk. During the match I mitigate without changing partitioning — kill stuck members, freeze autoscaling, raise downstream concurrency, or start a fresh consumer group from latest for the live dashboard while the old one catches up. After the match I fix the key design, move to cooperative or static membership, and alert on per-partition time lag.
A free account unlocks the rest of it, and saves your progress across every track.
strace Trace the system calls a process makesltrace Trace the library calls a program makeslsof List open files, sockets and the processes holding themdmesg Read the kernel's message logperf Profile where CPU time goes, kernel and user spacegdb Debug crashes, core dumps and hung processescoredumpctl Find and open core dumps captured by systemdsar Collect and replay system activity history