coredumpctlFind and open core dumps captured by systemd
coredumpctl lists the crashes that systemd-coredump has captured on this machine, shows the metadata and a ready-made stack trace for each, and opens any of them straight in gdb. It turns 'the service crashed overnight' into a backtrace in two commands.
coredumpctl [list|info|debug|dump] [-S SINCE] [-r] [-1] [-o FILE] [PID|COMM|EXE]On a fresh Ubuntu 24.04 server you want crashes of your services to be saved and easy to find.
sudoCores of system services are readable only by root (they contain the process's memory, including secrets).
When a process dies from a signal whose default action is to dump core (SIGSEGV, SIGABRT, SIGBUS, SIGQUIT...), the kernel checks kernel.core_pattern. With systemd-coredump installed it is a pipe: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h. The kernel starts that helper and streams the process's memory image into its stdin. systemd-coredump reads extra details from /proc/PID (command line, cgroup, unit, environment), generates a stack trace with elfutils, compresses the core (zstd) into /var/lib/systemd/coredump/ and writes a structured journal entry with all of it. coredumpctl simply queries the journal for those entries (MESSAGE_ID for coredumps) and, for debug, extracts the core to a temp file and runs gdb on it. Limits (MaxUse, ProcessSizeMax, Storage=external|journal|none) live in /etc/systemd/coredump.conf, and systemd-tmpfiles deletes old cores after 3 days by default. Ubuntu ships apport instead; installing systemd-coredump takes over core_pattern.
The kernel pipes the dying process's memory to systemd-coredump, which stores it and logs a journal entry that coredumpctl reads.
api-server dereferences a NULL pointer. The CPU faults, and the kernel sends SIGSEGV, whose default action is to terminate and dump core.
listTable of captured crashes: time, PID, UID, signal, core present?, executableinfo [MATCH]Details and stack trace of a crash (latest if no match)debug [MATCH]Open the core in gdb with the right executable (alias: gdb)dump -o FILE [MATCH]Write the core to a file, e.g. to analyse elsewhere-S / -U TIMESince / until a time: -S today, -S -2h-1Only the most recent matching entry-rReverse order: newest firstMATCHA PID, a command name (COMM), an executable path, or FIELD=VALUE--debugger-argumentsPass options to gdb, e.g. "-batch -ex 'bt full'"Running coredumpctl list without sudo and concluding cores weren't savedAs a normal user you see crashes of system services marked inaccessible, and you can't debug or dump them.
sudo coredumpctl listExpecting a core for a process killed with SIGKILL or SIGTERMOnly signals whose default action is 'core' (SEGV, ABRT, BUS, FPE, ILL, QUIT, TRAP, SYS) produce one. An OOM kill (SIGKILL) never does.
For OOM kills, check dmesg / journalctl -k; to snapshot a live process, use gcore PIDSetting LimitCORE=0 or leaving ulimit -c 0 in the unit and wondering why cores are missingsystemd-coredump honours the process's RLIMIT_CORE (%c in core_pattern): with 0 it logs the crash but stores no core.
LimitCORE=infinity in the service unitTry these on a Linux machine of your own. Each task has a hint and a solution.
Find out where this machine sends core dumps.
Make 'sleep 300' dump core on purpose and find it with coredumpctl (install systemd-coredump first if it's missing). In the practice sandbox the host decides where cores go (kernel.core_pattern) and coredumpctl isn't available, so write the commands; on an Ubuntu VM install systemd-coredump first.
Show only crashes from the last hour, newest first. In the practice sandbox the host decides where cores go (kernel.core_pattern) and coredumpctl isn't available, so write the commands; on an Ubuntu VM install systemd-coredump first.
coredumpctlFind and open core dumps captured by systemd
coredumpctl lists the crashes that systemd-coredump has captured on this machine, shows the metadata and a ready-made stack trace for each, and opens any of them straight in gdb. It turns 'the service crashed overnight' into a backtrace in two commands.
coredumpctl [list|info|debug|dump] [-S SINCE] [-r] [-1] [-o FILE] [PID|COMM|EXE]On a fresh Ubuntu 24.04 server you want crashes of your services to be saved and easy to find.
sudoCores of system services are readable only by root (they contain the process's memory, including secrets).
When a process dies from a signal whose default action is to dump core (SIGSEGV, SIGABRT, SIGBUS, SIGQUIT...), the kernel checks kernel.core_pattern. With systemd-coredump installed it is a pipe: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h. The kernel starts that helper and streams the process's memory image into its stdin. systemd-coredump reads extra details from /proc/PID (command line, cgroup, unit, environment), generates a stack trace with elfutils, compresses the core (zstd) into /var/lib/systemd/coredump/ and writes a structured journal entry with all of it. coredumpctl simply queries the journal for those entries (MESSAGE_ID for coredumps) and, for debug, extracts the core to a temp file and runs gdb on it. Limits (MaxUse, ProcessSizeMax, Storage=external|journal|none) live in /etc/systemd/coredump.conf, and systemd-tmpfiles deletes old cores after 3 days by default. Ubuntu ships apport instead; installing systemd-coredump takes over core_pattern.
The kernel pipes the dying process's memory to systemd-coredump, which stores it and logs a journal entry that coredumpctl reads.
api-server dereferences a NULL pointer. The CPU faults, and the kernel sends SIGSEGV, whose default action is to terminate and dump core.
listTable of captured crashes: time, PID, UID, signal, core present?, executableinfo [MATCH]Details and stack trace of a crash (latest if no match)debug [MATCH]Open the core in gdb with the right executable (alias: gdb)dump -o FILE [MATCH]Write the core to a file, e.g. to analyse elsewhere-S / -U TIMESince / until a time: -S today, -S -2h-1Only the most recent matching entry-rReverse order: newest firstMATCHA PID, a command name (COMM), an executable path, or FIELD=VALUE--debugger-argumentsPass options to gdb, e.g. "-batch -ex 'bt full'"Running coredumpctl list without sudo and concluding cores weren't savedAs a normal user you see crashes of system services marked inaccessible, and you can't debug or dump them.
sudo coredumpctl listExpecting a core for a process killed with SIGKILL or SIGTERMOnly signals whose default action is 'core' (SEGV, ABRT, BUS, FPE, ILL, QUIT, TRAP, SYS) produce one. An OOM kill (SIGKILL) never does.
For OOM kills, check dmesg / journalctl -k; to snapshot a live process, use gcore PIDSetting LimitCORE=0 or leaving ulimit -c 0 in the unit and wondering why cores are missingsystemd-coredump honours the process's RLIMIT_CORE (%c in core_pattern): with 0 it logs the crash but stores no core.
LimitCORE=infinity in the service unitTry these on a Linux machine of your own. Each task has a hint and a solution.
Find out where this machine sends core dumps.
Make 'sleep 300' dump core on purpose and find it with coredumpctl (install systemd-coredump first if it's missing). In the practice sandbox the host decides where cores go (kernel.core_pattern) and coredumpctl isn't available, so write the commands; on an Ubuntu VM install systemd-coredump first.
Show only crashes from the last hour, newest first. In the practice sandbox the host decides where cores go (kernel.core_pattern) and coredumpctl isn't available, so write the commands; on an Ubuntu VM install systemd-coredump first.