An Indian UPI payments app processing about 6,000 payments per second on a Saturday evening gets an alert: success rate for payments where the customer's account is at one large public-sector bank has fallen from 97 to 71 percent over fifteen minutes. That bank is about 14 percent of traffic. Overall success rate has dipped only from 96 to 92 percent, the app's own error rate and latency look normal, and there was a deploy of the payment orchestrator 40 minutes ago.
The interviewer asks“You're on call. Payment success for one large bank has dropped from 97 to 71 percent in fifteen minutes, every other bank looks normal. Walk me through what you do, in order.”
I'd first segment the drop: by NPCI response code class (business decline versus technical decline versus timeout), by whether the bank is the payer's bank, the payee's bank or our sponsor PSP, and by our pod version. That one breakdown points to the owner — the bank's core system, NPCI's link to the bank, our connectivity, or our deploy. While investigating, I mitigate: open a per-bank breaker or fail-fast prompt before PIN entry so users switch to another linked account, suppress retries while keeping status checks for in-flight payments, and post a status message. Then I escalate with evidence to the bank and NPCI, keep stakeholders updated on a fixed cadence, and follow up with per-bank, per-code alerting.
A free account unlocks the rest of it, and saves your progress across every track.
strace Trace the system calls a process makesltrace Trace the library calls a program makeslsof List open files, sockets and the processes holding themdmesg Read the kernel's message logperf Profile where CPU time goes, kernel and user spacegdb Debug crashes, core dumps and hung processescoredumpctl Find and open core dumps captured by systemdsar Collect and replay system activity history