A professional social network had a 47-minute SEV1 yesterday: a rate-limiter config change, pushed to all four data centres at once, contained a pattern that matched most members and throttled about 60 percent of message sends. It is the third config-push incident in the last eight SEV1 and SEV2 incidents. Previous reviews produced action items like 'add a second reviewer' and 'be more careful with regexes', and about 40 percent of action items from the last two quarters are still open.
The interviewer asks“You've been asked to run the post-incident review for yesterday's messaging outage. Our past reviews produce documents and action items that mostly never get done. How would you run this one so it leads to real, lasting change?”
I'd run it as a learning review, not a document exercise: a facilitator who wasn't involved, a timeline built from logs and one-to-one interviews before the meeting, and the question 'how did this make sense at the time' instead of 'who made the mistake'. I'd look for contributing factors across change safety, detection, response and organisation rather than a single root cause, and write action items that change systems, not people: staged config rollout and semantic validation instead of 'add a reviewer'. Each item gets a named owner, a date and a priority in the same backlog as roadmap work, with closure rates reviewed by leadership. And because this is the third config-push incident in eight, the most important output is a cross-incident theme that funds a programme to give config the same rollout safety as code.
A free account unlocks the rest of it, and saves your progress across every track.