CrackOps
Interview PrepTracksLinuxAWS LabSoonInterview RoomSoon
Sign inStart free
CrackOps

Interview prep for the people who keep production up.

[email protected]
ProductInterview PrepTracksDashboardInterview Room
RolesDevOpsSREPlatformCloud
CompanyAboutContactPrivacyTerms
© CrackOps
Linux / Debugging & Diagnostics / dmesg

dmesg

Read the kernel's message log

Beginnerdebuggingkernellogs

dmesg prints the kernel ring buffer: messages from the kernel and its drivers about hardware, disks, network cards, memory and crashes. When a process vanishes (OOM killer), a disk goes read-only or a program segfaults, the kernel's side of the story is here.

Syntaxdmesg [-T] [-H] [-w] [-l LEVEL[,LEVEL]...] [-f FACILITY] [-k] [-c]

Watch it run

You've logged in to a server and want to see what the kernel has been saying.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 3
dmesg

Ubuntu sets kernel.dmesg_restrict = 1: kernel messages can contain addresses and hardware details, so only root may read them.

Anatomy of a command

$
commandsudo

Ubuntu sets kernel.dmesg_restrict=1, so reading the kernel log needs root (CAP_SYSLOG).

How it works

The kernel writes its messages (printk) into a fixed-size in-memory ring buffer, typically 256 KB to 1 MB (log_buf_len= on the kernel command line changes it). Each record has a priority (emerg to debug), a facility, a sequence number and a timestamp in microseconds since boot. dmesg reads the records from /dev/kmsg, or with the older syslog(2) call, and formats them. Because the buffer is a ring, old messages are overwritten on a chatty system, and it is empty again after a reboot. systemd-journald also reads /dev/kmsg and stores the same messages, so journalctl -k -b -1 shows the kernel log of the previous boot if the journal is persistent. The default timestamp is seconds since boot; -T converts it to wall-clock time using the boot time, which drifts if the machine was suspended. kernel.dmesg_restrict=1 limits reading to root.

Options you'll actually use

-THuman-readable timestamps (can be off after suspend/resume)
-HHuman output: relative times, colours, pager
-wFollow: wait for and print new messages, like tail -f
-l LEVELSOnly these levels: emerg, alert, crit, err, warn, notice, info, debug
-xShow the facility and level as a prefix (kern :err :)
--notimeDon't print timestamps (shorter lines for grepping)
-kKernel messages only (skip messages written to /dev/kmsg from user space)
-cPrint, then clear the buffer (needs root; rarely a good idea)
--since / --untilTime window, e.g. --since '1 hour ago' (util-linux 2.38+)
journalctl -kSame messages from the journal; add -b -1 for the previous boot

Common mistakes

Checking dmesg after a reboot to find out why the server crashed

The ring buffer lives in RAM and starts empty at every boot, so the messages from before the crash are gone.

journalctl -k -b -1 (needs persistent journal: Storage=persistent or /var/log/journal)
Trusting -T timestamps to the second on a VM that was paused or a laptop that slept

-T adds seconds-since-boot to the boot time; time spent suspended isn't counted, so the clock drifts.

Compare with journalctl -k, which records real wall-clock time
sudo dmesg -c to 'clean up' before testing

-c clears the buffer for everyone: the next person debugging (or your monitoring) loses the evidence.

Note the last timestamp, or use dmesg -w / --since to see only new messages

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Try to read the kernel log as your normal user, then as root. Explain the difference.

  2. 2

    Show only error and warning messages from the kernel, with human-readable timestamps. In the practice sandbox dmesg says 'Operation not permitted': a container shares the host's kernel, and reading its log is blocked (dmesg_restrict, no CAP_SYSLOG). Write the command; run it on a real VM.

  3. 3

    Search the kernel log for any OOM killer or segfault events. In the practice sandbox dmesg says 'Operation not permitted': a container shares the host's kernel, and reading its log is blocked (dmesg_restrict, no CAP_SYSLOG). Write the command; run it on a real VM.

Related commands

journalctlQuery systemd's journal: service, boot and kernel logscoredumpctlFind and open core dumps captured by systemdfreeShow used, free and available memorymountAttach a filesystem to the directory tree
PreviouslsofNextperf
Linux / Debugging & Diagnostics / dmesg

dmesg

Read the kernel's message log

Beginnerdebuggingkernellogs

dmesg prints the kernel ring buffer: messages from the kernel and its drivers about hardware, disks, network cards, memory and crashes. When a process vanishes (OOM killer), a disk goes read-only or a program segfaults, the kernel's side of the story is here.

Syntaxdmesg [-T] [-H] [-w] [-l LEVEL[,LEVEL]...] [-f FACILITY] [-k] [-c]

Watch it run

You've logged in to a server and want to see what the kernel has been saying.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 3
dmesg

Ubuntu sets kernel.dmesg_restrict = 1: kernel messages can contain addresses and hardware details, so only root may read them.

Anatomy of a command

$
commandsudo

Ubuntu sets kernel.dmesg_restrict=1, so reading the kernel log needs root (CAP_SYSLOG).

How it works

The kernel writes its messages (printk) into a fixed-size in-memory ring buffer, typically 256 KB to 1 MB (log_buf_len= on the kernel command line changes it). Each record has a priority (emerg to debug), a facility, a sequence number and a timestamp in microseconds since boot. dmesg reads the records from /dev/kmsg, or with the older syslog(2) call, and formats them. Because the buffer is a ring, old messages are overwritten on a chatty system, and it is empty again after a reboot. systemd-journald also reads /dev/kmsg and stores the same messages, so journalctl -k -b -1 shows the kernel log of the previous boot if the journal is persistent. The default timestamp is seconds since boot; -T converts it to wall-clock time using the boot time, which drifts if the machine was suspended. kernel.dmesg_restrict=1 limits reading to root.

Options you'll actually use

-THuman-readable timestamps (can be off after suspend/resume)
-HHuman output: relative times, colours, pager
-wFollow: wait for and print new messages, like tail -f
-l LEVELSOnly these levels: emerg, alert, crit, err, warn, notice, info, debug
-xShow the facility and level as a prefix (kern :err :)
--notimeDon't print timestamps (shorter lines for grepping)
-kKernel messages only (skip messages written to /dev/kmsg from user space)
-cPrint, then clear the buffer (needs root; rarely a good idea)
--since / --untilTime window, e.g. --since '1 hour ago' (util-linux 2.38+)
journalctl -kSame messages from the journal; add -b -1 for the previous boot

Common mistakes

Checking dmesg after a reboot to find out why the server crashed

The ring buffer lives in RAM and starts empty at every boot, so the messages from before the crash are gone.

journalctl -k -b -1 (needs persistent journal: Storage=persistent or /var/log/journal)
Trusting -T timestamps to the second on a VM that was paused or a laptop that slept

-T adds seconds-since-boot to the boot time; time spent suspended isn't counted, so the clock drifts.

Compare with journalctl -k, which records real wall-clock time
sudo dmesg -c to 'clean up' before testing

-c clears the buffer for everyone: the next person debugging (or your monitoring) loses the evidence.

Note the last timestamp, or use dmesg -w / --since to see only new messages

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Try to read the kernel log as your normal user, then as root. Explain the difference.

  2. 2

    Show only error and warning messages from the kernel, with human-readable timestamps. In the practice sandbox dmesg says 'Operation not permitted': a container shares the host's kernel, and reading its log is blocked (dmesg_restrict, no CAP_SYSLOG). Write the command; run it on a real VM.

  3. 3

    Search the kernel log for any OOM killer or segfault events. In the practice sandbox dmesg says 'Operation not permitted': a container shares the host's kernel, and reading its log is blocked (dmesg_restrict, no CAP_SYSLOG). Write the command; run it on a real VM.

Related commands

journalctlQuery systemd's journal: service, boot and kernel logscoredumpctlFind and open core dumps captured by systemdfreeShow used, free and available memorymountAttach a filesystem to the directory tree
PreviouslsofNextperf