A professional social network's connections service (who is connected to whom) is backed by sharded MySQL: 1,024 logical shards on 64 primaries, about 30 TB, 150,000 reads and 12,000 writes a second. All primaries live in DC-1, whose lease ends in four months; the target is DC-3, 12 ms round trip away. Almost every product surface calls this service, and a binlog-based CDC stream feeds caches and search.
The interviewer asks“The connections store has to leave its current data centre within four months, with no downtime and no lost writes. Walk me through the migration plan phase by phase, how you prove the data is identical, and how you roll back at each step.”
I'd run it as five reversible phases. First, replicate: clone every shard into DC-3 and attach the clones as GTID replicas, throttled so the 30 TB copy doesn't saturate the inter-DC link. Second, verify: chunked checksums at a known GTID, sampled dual reads compared off the request path, and domain invariants, then move DC-3's reads to the local replicas. Third, shadow writes: mirror production writes to the new DC-3 write tier in dry-run mode, to prove capacity and that it computes identical mutations. Fourth, cut over write ownership shard by shard, canary first: buffer writes in the router, fence the old primary with super_read_only, wait for the target to apply the final GTID, bump the ownership epoch, start reverse replication and release the buffer, for about two seconds of added latency per shard and no errors. Fifth, keep DC-1 as a reverse replica for two weeks as the rollback path, then decommission it.
A free account unlocks the rest of it, and saves your progress across every track.