An Indian UPI payments app deployed a new version of its payment orchestrator at 14:00 on a sale day, using Argo Rollouts with a 10 percent canary for 10 minutes gated on error rate. Since about 14:20 the success rate has decayed steadily rather than dropping in a step. No alerts fired on CPU or memory, and partner dashboards show no declared incident.
The interviewer asks“The deploy went out clean, canary passed, and for twenty minutes everything looked perfect. Over the next hour payment success has slid from 99.2 to 96.8 percent and it's still dropping. What do you suspect, what do you do right now, and why didn't the canary catch it?”
A slow decay after a clean deploy means something accumulates or expires: a leak in connections, file descriptors or threads, a token or cache that expires after its TTL, retries piling up, or a partner rate limit reached as volume grows. On a sale day with the deploy as the only change, I roll back now, without a root cause, after grabbing evidence from two bad pods. Then I confirm with a discriminating test, like a restarted pod on the new version being perfect and then decaying again, and the metric that matches the curve, such as pool pending threads or open file descriptors. The canary missed it because ten minutes was shorter than the decay and it gated on HTTP error rate instead of payment success by version.
A free account unlocks the rest of it, and saves your progress across every track.
bash: variables Shell variables: assign, quote, default and exportbash: if Conditions: if, elif, else, test brackets and casebash: loops Loops: for, while read, until and retrybash: functions Functions: name a block, pass arguments, return a statusbash: arrays Arrays: lists and key-value maps in bashbash: exit codes Exit codes, $?, set -euo pipefail and trapbash: redirection Redirection: stdout, stderr, 2>&1, /dev/null and here-docsbash: arguments Script arguments: $1, $#, "$@", shift and getopts