A regulated broking platform runs its order-entry gateway as 40 JVM pods on Kubernetes, 2 vCPU limit each, in front of a risk-check service and Postgres. Inbound order rate goes from about 300 per second in pre-open to 25,000 per second within a second of 09:15. Order-ack p99 is normally 4 ms; from 09:15:00 to roughly 09:16:30 it hits 180 ms and p99.9 touches 900 ms, while the average only moves from 3 ms to 6 ms. The HPA scales up, but only around 09:17.
The interviewer asks“Every trading day, the moment the market opens at 09:15, our order-entry p99 jumps for about ninety seconds and then settles. Everyone has a theory. How would you investigate this, and how would you prove the cause to the point where an auditor or the CTO would accept it?”
I treat it as a measurement problem first: per-second percentiles from histograms, synchronised clocks, and every signal aligned on one 09:10 to 09:20 timeline. Then I rank hypotheses for a cold start into a step load: CFS throttling, JIT and cache warm-up, idle connections killed overnight, GC, accept-queue overflows, cron jobs, and an autoscaler that reacts two minutes late, each with the one signal that confirms or kills it. I reproduce it with an open-model load test that replays the real market-open arrival curve against cold pods, then prove the fix with a same-day cohort experiment, half the pods pre-warmed and half not, repeated over several trading days. The evidence pack is hypothesis, method, numbers and residual risk, so nobody has to take my word for it.
A free account unlocks the rest of it, and saves your progress across every track.
sar Collect and replay system activity historympstat Per-CPU utilisation: find the hot corepidstat Per-process CPU, memory, disk and context switches over timesysctl Read and tune kernel parameters at runtimeulimit Show and set per-process resource limitsnumactl Control NUMA CPU and memory placementstrace Trace the system calls a process makesltrace Trace the library calls a program makes