CrackOps
Interview PrepTracksLinuxAWS LabSoonInterview RoomSoon
Sign inStart free
CrackOps

Interview prep for the people who keep production up.

[email protected]
ProductInterview PrepTracksDashboardInterview Room
RolesDevOpsSREPlatformCloud
CompanyAboutContactPrivacyTerms
© CrackOps
Tracks/Production Incidents
Expert 4 minTroubleshooting

Playback failures spike for 3% of APAC viewers while CPU, memory and pods look fine. How do you triage it?

Scenario

A live-sports streaming platform based in India also streams to licensed APAC markets such as Singapore, Malaysia and Australia, about 2.5 million concurrent viewers outside India during a big match. Video goes out through two CDNs with HLS and DASH, playback and DRM services run on Kubernetes in Indian regions, and every player sends QoE beacons with error codes. Nothing was deployed by the platform team today, but the CDNs, ISPs, app stores and data vendors change things without telling anyone.

The interviewer asks

“Twenty minutes into a match, the client QoE dashboard shows video start failures for APAC viewers jumping from 0.4 percent to 3.1 percent. CPU, memory, pod health and server-side 5xx all look normal, and the service teams say it is not them. Walk me through how you find what the 3 percent have in common, stop the bleeding, and then get to root cause.”

The 30-second answer

Healthy servers tell me the failure is before the origin, on the client, or a 4xx that 5xx dashboards ignore, so I stop staring at pods and slice the failing sessions. From the QoE beacons I compute lift: share of failures divided by share of sessions, across ASN, CDN and PoP, IP version, device and app version, DRM system, manifest variant and error code. When a cohort stands out, for example one mobile ISP on IPv6 through one CDN PoP, I take the cheapest reversible mitigation first, such as steering that ISP to the other CDN, and confirm recovery in the same RUM data. Then I prove root cause with probes inside that network and CDN edge logs, and prevent recurrence with sliced RUM SLOs, IPv4 and IPv6 synthetic probes and automatic QoE-based CDN steering.

Sign in to finish this question

A free account unlocks the rest of it, and saves your progress across every track.

  • The voiced mock interview (4 min)
  • The full senior answer, step by step
  • Animated architecture diagrams
  • The whole conversation, 14 turns
  • Deep dive: how it works, mistakes, risks and trade-offs
  • 3 copy-ready command snippets
  • 5 follow-up questions they'll ask next
  • A self-check of 8 things interviewers listen for
Create a free accountSign in

More Production Incidents interview questions

  • The end-of-day settlement batch used to finish by 06:00 and now ends at 09:00. How do you find out why?
  • The database failed over, but the application keeps talking to the old primary. What do you do?
  • Kafka lag hits two minutes during a surge, but the consumers are idle. Where do you look?
  • p99 latency on the Redis session store jumps 40x mid-match while p50 is fine. How do you find the cause?
  • Payment success rate has dropped for one bank only. How do you find out whose problem it is and protect users?
  • A payment partner slows down and your own retries turn it into an outage. How do you stop the storm?
All Production Incidents questions

Linux commands for this track

  • strace Trace the system calls a process makes
  • ltrace Trace the library calls a program makes
  • lsof List open files, sockets and the processes holding them
  • dmesg Read the kernel's message log
  • perf Profile where CPU time goes, kernel and user space
  • gdb Debug crashes, core dumps and hung processes
  • coredumpctl Find and open core dumps captured by systemd
  • sar Collect and replay system activity history