An Indian UPI payments app sends payments through a PSP adapter service that calls one sponsor bank's switch. The call path is app to API gateway to orchestrator to PSP adapter, all on Kubernetes with an Istio mesh. At 8:10 p.m. on a sale day the bank's p99 rose from 300 ms to 4 s; within three minutes calls to the bank went from 9,000 to 41,000 per second, the orchestrator's thread pools were exhausted, and payments routed through the other two healthy sponsor banks also started failing.
The interviewer asks“A sponsor bank's latency went from 300 milliseconds to 4 seconds, and now our request rate to them is several times normal and everything is timing out, including payments to healthy banks. Walk me through how you stop it right now, and then how you'd make sure it can't happen again.”
First I stop the amplification: open the breaker for the slow sponsor bank and steer new payments to the healthy two, switch off retries at every layer except one, and cap concurrency to that bank so its slowness can't consume shared threads. For payments specifically, a timeout becomes PENDING and is resolved by status check on the same transaction ID, never by re-sending. The math explains the urgency: three layers each making three attempts is up to 27x load, and a 4-second latency alone needs 13x the concurrency by Little's law. Long term: one retry layer with exponential backoff and full jitter, retry budgets, deadlines that shrink as they propagate, per-partner bulkheads and load shedding before saturation.
A free account unlocks the rest of it, and saves your progress across every track.
strace Trace the system calls a process makesltrace Trace the library calls a program makeslsof List open files, sockets and the processes holding themdmesg Read the kernel's message logperf Profile where CPU time goes, kernel and user spacegdb Debug crashes, core dumps and hung processescoredumpctl Find and open core dumps captured by systemdsar Collect and replay system activity history