gdbDebug crashes, core dumps and hung processes
gdb, the GNU debugger, lets you stop a program and look inside it: the call stack, variables and memory at the moment it crashed. For a DevOps engineer its main jobs are reading a core dump after a segfault and getting stack traces from a hung or deadlocked process.
gdb [-q] [-batch] [-ex CMD]... PROGRAM [CORE] | gdb -p PID | gdb --args PROGRAM [ARG]...A small C tool you build segfaults on start. You want the exact line and the call path.
gdbThe GNU debugger. Controls the process through ptrace, like strace, but stops it completely.
gdb uses ptrace to control a live process: attach (PTRACE_ATTACH or as parent), stop it, read and write its registers and memory, and set breakpoints by replacing an instruction with int3. To make sense of raw memory it reads the program's symbol table and DWARF debug information (from the binary built with -g, or from separate debug files in /usr/lib/debug or via debuginfod), which map addresses to function names, source lines, variables and types. A core file is a snapshot of a dead process's memory and registers written by the kernel when it was killed by a signal like SIGSEGV; gdb loads it together with the original executable and the same shared libraries, so you can inspect the crash after the fact without the process running. Without debug info you still get function names from exported symbols, but no line numbers or variables. Attaching stops all threads until you detach.
gdb PROG COREOpen a core dump with the binary that produced itgdb -p PIDAttach to a running process (stops it until you detach or quit)gdb --args PROG ARGSLoad a program with its arguments, then type run-batch -ex CMDRun commands non-interactively and exit: good for scripts and ticketsbt / bt fullBacktrace: the call stack (full adds local variables)thread apply all btBacktrace of every thread: finds deadlocksframe N / up / downSelect a stack frame to inspectprint EXPR / info localsShow a variable or expression / all locals in the framebreak FILE:LINE / FUNCSet a breakpoint, then run, next, step, continueinfo sharedlibraryWhich libraries are loaded, and whether their symbols were founddetach / quitLet the process continue / leave gdbExpecting a core file after a crash, but ulimit -c is 0With a core size limit of 0 the kernel writes nothing. Many shells and services start with that limit.
ulimit -c unlimited for the shell, LimitCORE=infinity in the systemd unit, and check kernel.core_patternOpening a core with a newer build of the binaryAddresses no longer match the code, so gdb prints wrong functions or garbage ('core file may not match specified executable file').
Keep the exact binary and debug symbols for every release (build ID, debuginfod)Attaching gdb interactively to a production process and going for coffeeEvery thread is stopped while gdb is attached: requests time out and health checks fail.
gdb -batch -ex 'thread apply all bt' -p PID (in and out in a second), or gcore PID to snapshot itDebugging an optimised (-O2) build and trusting every variable valueOptimised code keeps values in registers or removes them: gdb shows <optimized out> or stale values.
Reproduce with -O0 -g when you can, or reason from the backtrace and registersTry these on a Linux machine of your own. Each task has a hint and a solution.
Write a 3-line C program that dereferences a NULL pointer, compile it with -g, and find the crashing line with gdb.
Make the same program leave a core dump in the current directory, then open it in gdb.
Start 'sleep 600 &' and print its backtrace with gdb without stopping it for more than a moment.
gdbDebug crashes, core dumps and hung processes
gdb, the GNU debugger, lets you stop a program and look inside it: the call stack, variables and memory at the moment it crashed. For a DevOps engineer its main jobs are reading a core dump after a segfault and getting stack traces from a hung or deadlocked process.
gdb [-q] [-batch] [-ex CMD]... PROGRAM [CORE] | gdb -p PID | gdb --args PROGRAM [ARG]...A small C tool you build segfaults on start. You want the exact line and the call path.
gdbThe GNU debugger. Controls the process through ptrace, like strace, but stops it completely.
gdb uses ptrace to control a live process: attach (PTRACE_ATTACH or as parent), stop it, read and write its registers and memory, and set breakpoints by replacing an instruction with int3. To make sense of raw memory it reads the program's symbol table and DWARF debug information (from the binary built with -g, or from separate debug files in /usr/lib/debug or via debuginfod), which map addresses to function names, source lines, variables and types. A core file is a snapshot of a dead process's memory and registers written by the kernel when it was killed by a signal like SIGSEGV; gdb loads it together with the original executable and the same shared libraries, so you can inspect the crash after the fact without the process running. Without debug info you still get function names from exported symbols, but no line numbers or variables. Attaching stops all threads until you detach.
gdb PROG COREOpen a core dump with the binary that produced itgdb -p PIDAttach to a running process (stops it until you detach or quit)gdb --args PROG ARGSLoad a program with its arguments, then type run-batch -ex CMDRun commands non-interactively and exit: good for scripts and ticketsbt / bt fullBacktrace: the call stack (full adds local variables)thread apply all btBacktrace of every thread: finds deadlocksframe N / up / downSelect a stack frame to inspectprint EXPR / info localsShow a variable or expression / all locals in the framebreak FILE:LINE / FUNCSet a breakpoint, then run, next, step, continueinfo sharedlibraryWhich libraries are loaded, and whether their symbols were founddetach / quitLet the process continue / leave gdbExpecting a core file after a crash, but ulimit -c is 0With a core size limit of 0 the kernel writes nothing. Many shells and services start with that limit.
ulimit -c unlimited for the shell, LimitCORE=infinity in the systemd unit, and check kernel.core_patternOpening a core with a newer build of the binaryAddresses no longer match the code, so gdb prints wrong functions or garbage ('core file may not match specified executable file').
Keep the exact binary and debug symbols for every release (build ID, debuginfod)Attaching gdb interactively to a production process and going for coffeeEvery thread is stopped while gdb is attached: requests time out and health checks fail.
gdb -batch -ex 'thread apply all bt' -p PID (in and out in a second), or gcore PID to snapshot itDebugging an optimised (-O2) build and trusting every variable valueOptimised code keeps values in registers or removes them: gdb shows <optimized out> or stale values.
Reproduce with -O0 -g when you can, or reason from the backtrace and registersTry these on a Linux machine of your own. Each task has a hint and a solution.
Write a 3-line C program that dereferences a NULL pointer, compile it with -g, and find the crashing line with gdb.
Make the same program leave a core dump in the current directory, then open it in gdb.
Start 'sleep 600 &' and print its backtrace with gdb without stopping it for more than a moment.