CrackOps
Interview PrepTracksLinuxAWS LabSoonInterview RoomSoon
Sign inStart free
CrackOps

Interview prep for the people who keep production up.

[email protected]
ProductInterview PrepTracksDashboardInterview Room
RolesDevOpsSREPlatformCloud
CompanyAboutContactPrivacyTerms
© CrackOps
Tracks/SRE & Observability
Advanced 4 minScenario based

How would you monitor HPA scaling decisions in real time and catch metrics-server or adapter lag before it hurts?

Scenario

A live-sports streaming platform runs about 150 HPAs per cluster. CPU-based HPAs read from metrics-server; the heartbeat and playback services scale on requests per second per pod served by prometheus-adapter from a shared Prometheus. In the semi-final, concurrency jumped from 18 to 29 million in six minutes; the heartbeat HPA stayed at 140 replicas for four minutes while p99 latency climbed to 3 seconds. The post-incident timeline showed Prometheus rule evaluation falling behind at peak.

The interviewer asks

“During the semi-final our heartbeat service did not scale for about four minutes during a join wave, and every HPA condition looked healthy. How would you monitor HPA decisions in real time across the fleet, and how would you detect that the metrics feeding the HPA are late or stale?”

The 30-second answer

I watch three layers. Decisions: kube-state-metrics series for desired, current, min and max replicas, the observed metric against its target, and the HPA conditions, with every rescale event shipped to the log store and drawn as annotations. Delivery: Deployment spec versus available replicas, so Pending and unready pods are visible. And, most importantly, input freshness: the healthy pipeline costs about 60 to 75 seconds, so I alert on sample age of the scaling series, on missed rule evaluations, on the APIService, and on an end-to-end canary metric read back through the custom metrics API. In the semi-final the adapter served a stale but valid value, so conditions stayed green; freshness checks catch that, and making stale values absent plus a CPU backstop or KEDA fallback turns a silent freeze into a loud, safe failure.

Sign in to finish this question

A free account unlocks the rest of it, and saves your progress across every track.

  • The voiced mock interview (4 min)
  • The full senior answer, step by step
  • Animated architecture diagrams
  • The whole conversation, 14 turns
  • Deep dive: how it works, mistakes, risks and trade-offs
  • 3 copy-ready command snippets
  • 5 follow-up questions they'll ask next
  • A self-check of 8 things interviewers listen for
Create a free accountSign in

More SRE & Observability interview questions

  • Order latency spikes every trading day at 09:15. How do you find the cause and prove it?
  • How would you run chaos experiments on a live streaming pipeline without viewers ever noticing?
  • Build a capacity plan for festival peaks you can predict and flash spikes you can't. Show me the math.
  • How would you roll out ownership, SLOs and error budgets across thousands of microservices?
All SRE & Observability questions

Linux commands for this track

  • sar Collect and replay system activity history
  • mpstat Per-CPU utilisation: find the hot core
  • pidstat Per-process CPU, memory, disk and context switches over time
  • sysctl Read and tune kernel parameters at runtime
  • ulimit Show and set per-process resource limits
  • numactl Control NUMA CPU and memory placement
  • strace Trace the system calls a process makes
  • ltrace Trace the library calls a program makes