An Indian UPI payments app runs its payment APIs on Amazon EKS with the Cluster Autoscaler on managed node groups and HPAs on CPU at 70 percent. On a major festival evening traffic ramps 9x in twenty minutes. p99 latency for the pay API has gone from 400 milliseconds to 6 seconds, success rate is falling, and 140 pods are Pending.
The interviewer asks“It's 7 pm on a festival day, payment traffic has gone from the usual 2,000 to 18,000 requests per second in twenty minutes, and pods are pending while latency climbs. Walk me through your first fifteen minutes, and then tell me why autoscaling lagged and what you'd change.”
First I protect the payment core: turn on admission control at the gateway and shed non-critical features like rewards, history and recommendations so the pay path gets the capacity we have. In parallel I scale by hand instead of waiting: raise HPA minReplicas for the pay services and the node group minimums, after checking the DB connection budget. Then I find what is actually limiting new capacity from the scheduler events: node provisioning time, image pulls, vCPU quotas or subnet IP exhaustion with the VPC CNI. Afterwards, the root cause is usually a reactive CPU-based HPA stacked on slow nodes and slow pod startup, so the fix is scheduled pre-scaling for known festivals, scaling on request rate with KEDA, and overprovisioning pods for warm headroom.
A free account unlocks the rest of it, and saves your progress across every track.
kubectl get List Kubernetes resources and their statuskubectl describe Explain an object's state, conditions and eventskubectl logs Read and follow container logskubectl exec Run a command or shell inside a running containerkubectl apply Create or update resources from YAML, declarativelykubectl delete Delete Kubernetes resources safelykubectl scale Change the replica count of a workloadkubectl rollout Watch, undo and restart Deployment rollouts