A professional social network with hundreds of millions of members ships thousands of microservices through an in-house deployment system. Every release goes through automatic canary analysis: 1 percent of hosts for 60 minutes, compared against the rest of the fleet on CPU, p99 latency and 5xx rate, then a 6-hour ramp to 100 percent. Release 2026.10.26 of feed-mixer scored 97 out of 100 and was fully deployed by 18:00 UTC. The next evening, the engagement dashboard shows feed sessions with at least one interaction down 1.9 percent week over week, with Android down 4 percent.
The interviewer asks“The feed service release passed canary analysis with a high score and rolled out to 100 percent. Thirty hours later, product analytics show feed interactions are down about 2 percent, mostly on Android. Walk me through what the canary could have missed, how you'd confirm which, and what you'd change in the canary process.”
A canary can only catch what it measures, on the traffic it receives, within the time it waits, with enough statistical power. So I first look at the shape of the drop against the rollout timeline and segment by app version, DC and member cohort, because that tells me which of those four was the blind spot. Here, Android-heavy and growing with the ramp points to a successful-but-wrong response that old clients render badly, which a CPU, latency and 5xx canary can't see. I'd roll back or flag it off to confirm, then fix the process: correctness and client metrics in the canary, member-sticky stages that bake through a daily peak, power-sized stages, and contract checks against live client versions.
A free account unlocks the rest of it, and saves your progress across every track.
bash: variables Shell variables: assign, quote, default and exportbash: if Conditions: if, elif, else, test brackets and casebash: loops Loops: for, while read, until and retrybash: functions Functions: name a block, pass arguments, return a statusbash: arrays Arrays: lists and key-value maps in bashbash: exit codes Exit codes, $?, set -euo pipefail and trapbash: redirection Redirection: stdout, stderr, 2>&1, /dev/null and here-docsbash: arguments Script arguments: $1, $#, "$@", shift and getopts