A live-sports streaming platform's pipeline runs from venue ingest through encoders, packagers and origin to a CDN, with playback APIs on Kubernetes backed by Redis and telemetry flowing through Kafka. Last season a single packager pod crash froze manifests for 40 seconds during a semi-final, and nobody had ever tested that failure. The tournament starts in six weeks.
The interviewer asks“Leadership wants confidence in the streaming pipeline before the tournament, but says no viewer may notice an experiment. How would you design and run chaos experiments safely? Walk me through one experiment in detail.”
I treat chaos as hypothesis testing with a seatbelt. Each experiment states a viewer-facing steady state — stall ratio, manifest age, start time — and a falsifiable prediction, then climbs a ladder: staging with a looped real feed and synthetic viewers, then production on a test channel or cell that only synthetic viewers watch, using header-scoped faults so real requests are untouched, and only then small real-traffic experiments and game days. Abort conditions are written before the run and wired to automation that removes the fault in seconds. I never run on match days, during incidents, or with a low error budget, and every finding becomes a fix plus a regression experiment.
A free account unlocks the rest of it, and saves your progress across every track.
sar Collect and replay system activity historympstat Per-CPU utilisation: find the hot corepidstat Per-process CPU, memory, disk and context switches over timesysctl Read and tune kernel parameters at runtimeulimit Show and set per-process resource limitsnumactl Control NUMA CPU and memory placementstrace Trace the system calls a process makesltrace Trace the library calls a program makes