A broker's order management, risk and post-trade ledger run in a primary data centre, with a second site 60 km away and a cloud region 1,400 km away. The ledger is PostgreSQL at about 4 TB with 6,000 writes per second at peak; the order core uses an in-memory replicated journal. Last year's DR test took 3 hours 40 minutes and lost 11 minutes of trades that had to be rebuilt from exchange files.
The interviewer asks“The regulator expects us to recover from the loss of our primary data centre in under 15 minutes with effectively no data loss. Design the DR setup, walk me through the recovery minute by minute, and tell me honestly what 'near zero' means.”
I'd make RPO zero true only where physics allows it: synchronous quorum commit from the primary to the metro site 60 km away, which costs about 0.6 ms per commit, and asynchronous replication to the far cloud region for a regional disaster, where RPO is honestly a second or less and is closed by reconciling with exchange drop copies. Failover is fenced and quorum-based with Patroni and a three-site etcd, the metro site is a hot standby with the application already running, and a person declares site failover within five minutes against fixed criteria. The runbook is timed to the minute, rehearsed quarterly, and once a year we run a full trading day from the DR site. Point-in-time recovery covers the case replication can't: logical corruption.
A free account unlocks the rest of it, and saves your progress across every track.