A professional social network runs thousands of microservices across four data centres: US-East, US-West, Europe and Singapore. Kafka is the backbone, with a local cluster in each DC and aggregate clusters fed by mirroring. The in-house traffic tool drains a DC by shifting GeoDNS and edge-to-origin weights, and reports progress as request rate at the DC's ingress compared with the pre-drain baseline. US-East normally carries about 35 percent of global traffic; the drain began at 20:00 UTC, moved quickly to 80 percent, and has not moved since 20:25.
The interviewer asks“We're draining a data centre ahead of a power maintenance, and the drain dashboard has sat at 80 percent for forty minutes. The maintenance starts in two hours. How do you work out what the remaining 20 percent is, and how do you finish the drain without causing an incident somewhere else?”
First I'd check whether the drain tool stopped itself, because the usual blocker is a capacity guard in the destination DCs, not stubborn clients. Then I'd attribute the remaining 20 percent by caller, protocol and connection age, using mesh telemetry, Envoy connection stats, DNS query logs and Kafka group membership. It typically falls into a few buckets: long-lived gRPC and websocket connections, clients caching DNS or with hardcoded endpoints, stateful leaders and primaries, pinned batch jobs and local Kafka consumers. Each needs its own fix: create headroom, rotate connections gradually, transfer leadership, drain Kafka to zero lag. Only a small, known, retry-safe remainder gets a hard cut, with abort criteria agreed in advance.
A free account unlocks the rest of it, and saves your progress across every track.
strace Trace the system calls a process makesltrace Trace the library calls a program makeslsof List open files, sockets and the processes holding themdmesg Read the kernel's message logperf Profile where CPU time goes, kernel and user spacegdb Debug crashes, core dumps and hung processescoredumpctl Find and open core dumps captured by systemdsar Collect and replay system activity history