A live-sports streaming platform runs about 150 HPAs per cluster. CPU-based HPAs read from metrics-server; the heartbeat and playback services scale on requests per second per pod served by prometheus-adapter from a shared Prometheus. In the semi-final, concurrency jumped from 18 to 29 million in six minutes; the heartbeat HPA stayed at 140 replicas for four minutes while p99 latency climbed to 3 seconds. The post-incident timeline showed Prometheus rule evaluation falling behind at peak.
The interviewer asks“During the semi-final our heartbeat service did not scale for about four minutes during a join wave, and every HPA condition looked healthy. How would you monitor HPA decisions in real time across the fleet, and how would you detect that the metrics feeding the HPA are late or stale?”
I watch three layers. Decisions: kube-state-metrics series for desired, current, min and max replicas, the observed metric against its target, and the HPA conditions, with every rescale event shipped to the log store and drawn as annotations. Delivery: Deployment spec versus available replicas, so Pending and unready pods are visible. And, most importantly, input freshness: the healthy pipeline costs about 60 to 75 seconds, so I alert on sample age of the scaling series, on missed rule evaluations, on the APIService, and on an end-to-end canary metric read back through the custom metrics API. In the semi-final the adapter served a stale but valid value, so conditions stayed green; freshness checks catch that, and making stale values absent plus a CPU backstop or KEDA fallback turns a silent freeze into a loud, safe failure.
A free account unlocks the rest of it, and saves your progress across every track.
sar Collect and replay system activity historympstat Per-CPU utilisation: find the hot corepidstat Per-process CPU, memory, disk and context switches over timesysctl Read and tune kernel parameters at runtimeulimit Show and set per-process resource limitsnumactl Control NUMA CPU and memory placementstrace Trace the system calls a process makesltrace Trace the library calls a program makes