An Indian UPI payments app processes about 4,000 transactions per second on a festival evening. Its ledger runs on PostgreSQL 16 with Patroni, one primary and two streaming replicas. At 20:41:07 IST the primary's host failed, Patroni promoted a replica in 31 seconds, and the apps reconnected. During the window, clients saw timeouts and some retried.
The interviewer asks“The primary ledger database just failed over while payments were in flight. Services are back up. How do you prove to me, the CFO and the bank partners that no transaction was lost and none was applied twice?”
I start from what the replication mode guaranteed: with async replication, acknowledged commits after the promoted replica's last replayed LSN may be gone, so I fence the old primary and preserve its WAL before anyone rewinds it. I compute the in-flight window from the timeline switch point, decode the old primary's WAL past that point, and diff it against the new primary. Then I reconcile three sources for every transaction in the window: our ledger, our outbox of NPCI requests, and NPCI's own status via transaction status check. Idempotency keys catch retries whose first attempt survived; the dangerous case is a retry whose first attempt was lost but reached NPCI, so I match on the UPI transaction ID and RRN, not just our keys. Finally I run invariant checks, like every transaction's ledger legs summing to zero, and only then close the incident.
A free account unlocks the rest of it, and saves your progress across every track.
strace Trace the system calls a process makesltrace Trace the library calls a program makeslsof List open files, sockets and the processes holding themdmesg Read the kernel's message logperf Profile where CPU time goes, kernel and user spacegdb Debug crashes, core dumps and hung processescoredumpctl Find and open core dumps captured by systemdsar Collect and replay system activity history