A broker's end-of-day batch starts at 21:30 after trade capture closes: about 180 jobs in an Airflow DAG covering trade enrichment, position revaluation, margin, fees, settlement instructions and regulatory reports, mostly against a PostgreSQL 15 database. Over eight weeks the finish time drifted from 06:00 to 09:10; the settlement file must reach the clearing house by 07:00, and ops have been sending it late with manual approvals. Trade volume is up about 30 percent over the same period, and nobody changed the batch code in the last month.
The interviewer asks“Our end-of-day settlement batch used to finish around 06:00 and now regularly finishes after 09:00, which is past the clearing house deadline. Walk me through how you'd find the bottleneck, what you'd do today, and how you'd stop it happening again.”
First I protect tonight: identify the critical path, defer reports that aren't on it, move anything competing for the database out of the window, and agree a manual fallback with ops. Then I pull eight weeks of per-task timings and row counts from the scheduler and compare each step's growth with volume growth: linear steps should be up about 30 percent, so a step up threefold points to a plan regression, lock waits or contention, not volume. I confirm with pg_stat_statements, auto_explain plans, wait events and I/O metrics at the hour that job runs. Then I fix the specific cause and add a completion SLO with an alert on projected finish time, so we know at 01:00, not 07:00.
A free account unlocks the rest of it, and saves your progress across every track.
strace Trace the system calls a process makesltrace Trace the library calls a program makeslsof List open files, sockets and the processes holding themdmesg Read the kernel's message logperf Profile where CPU time goes, kernel and user spacegdb Debug crashes, core dumps and hung processescoredumpctl Find and open core dumps captured by systemdsar Collect and replay system activity history