A professional social network runs about 3,000 services owned by 400 teams, across six data centres in two regions. Teams ship around 1,500 production deploys a day. Today each team uses its own scripts and eyeballs dashboards during rollouts. Two recent outages were bad deploys that went global within 20 minutes, one of them during a holiday change freeze.
The interviewer asks“Design the deployment system for the whole estate: how a change moves from one instance to every data centre, how the system decides on its own that a canary is bad, and how it rolls back. Tell me how this system behaves during an incident or a change freeze.”
I'd build a central deploy orchestrator, a durable workflow per deployment, that every service uses through a declarative manifest, with thin agents in each data centre doing the actual rollout. Every change moves in waves: a canary next to a fresh baseline of the old version in one DC, then that whole DC, then the rest of the region, then the other region, with bake times between waves. An automated judge compares canary and baseline per metric with a Mann-Whitney test and tolerated effect sizes, scores the result, and rolls back automatically on failure. Bake time is derived from traffic so each test has enough samples. A policy engine checks freezes, active incidents and error-budget state before every step, and there's a global halt switch and dependency-aware concurrency limits. The deploy system itself is built not to depend on anything it deploys.
A free account unlocks the rest of it, and saves your progress across every track.
bash: variables Shell variables: assign, quote, default and exportbash: if Conditions: if, elif, else, test brackets and casebash: loops Loops: for, while read, until and retrybash: functions Functions: name a block, pass arguments, return a statusbash: arrays Arrays: lists and key-value maps in bashbash: exit codes Exit codes, $?, set -euo pipefail and trapbash: redirection Redirection: stdout, stderr, 2>&1, /dev/null and here-docsbash: arguments Script arguments: $1, $#, "$@", shift and getopts