A broking platform's FIX gateway runs on a RHEL 9 VM with a single 200 GB /var filesystem holding the FIX session store (sequence numbers), FIX message logs that must be retained for regulatory purposes, application logs and journald. At 10:21 a static 90 percent alert fired; at 10:40 /var hit 100 percent and the gateway started logging write errors while the exchange sessions are still up. A change the previous evening switched one logger to DEBUG.
The interviewer asks“The filesystem on our exchange-connectivity host just hit 100 percent in the middle of the trading session. Talk me through exactly what you'd do in the first ten minutes, how you'd recover space without breaking anything, and what you'd change so it can't happen again.”
First I protect the session: confirm what is full, space or inodes, and how fast it is growing, because the rate tells me how many minutes I have. The fastest safe win on a critical host is often growing the volume online; otherwise I truncate, not delete, the runaway DEBUG log, release deleted-but-open files through /proc, vacuum journald, and move compressed regulated logs to archive, never deleting records, database files or WAL. Then I fix the trigger, revert the log level, and verify the FIX sessions and sequence numbers are healthy. Prevention is separate volumes for logs and the session store, size-capped rotation, journald and core-dump limits, off-host archival, and alerts on time-to-full and inodes rather than a static percentage.
A free account unlocks the rest of it, and saves your progress across every track.