A live-sports streaming platform keeps viewer sessions and device limits in a 12-shard Redis Cluster (each shard a primary plus a replica, about 22 GB per primary). At the innings break of a cricket final, with 28 million concurrent viewers, traffic is 1.3M ops/s. p50 stays at 0.3 ms, but p99 has gone from 1 ms to 45 ms and p999 to 400 ms; the playback API's session checks time out and clients start retrying.
The interviewer asks“Median latency is unchanged but p99 and p999 have exploded, and auth checks are timing out. Walk me through how you find what's blocking Redis, how you stop the bleeding, and what you change afterwards.”
A fine median with an exploding tail means requests are queueing behind something on Redis's single-threaded event loop, so I look for what's holding the loop. First I check whether it's one shard or all of them, then SLOWLOG and commandstats for O(N) commands on big keys, latest_fork_usec and the latency monitor for fork stalls, expired and evicted keys for expiry or eviction storms, and connected clients for a reconnect storm. I mitigate by removing the blocking work — kill the offending caller, pause BGSAVE, rate-limit reconnects — not by resharding mid-match. Afterwards I fix the data model, jitter TTLs, set THP to never, move persistence off primaries, and alert on per-shard p99.
A free account unlocks the rest of it, and saves your progress across every track.
strace Trace the system calls a process makesltrace Trace the library calls a program makeslsof List open files, sockets and the processes holding themdmesg Read the kernel's message logperf Profile where CPU time goes, kernel and user spacegdb Debug crashes, core dumps and hung processescoredumpctl Find and open core dumps captured by systemdsar Collect and replay system activity history