CrackOps
Interview PrepTracksLinuxAWS LabSoonInterview RoomSoon
Sign inStart free
CrackOps

Interview prep for the people who keep production up.

[email protected]
ProductInterview PrepTracksDashboardInterview Room
RolesDevOpsSREPlatformCloud
CompanyAboutContactPrivacyTerms
© CrackOps
Linux / Packages & Services / journalctl

journalctl

Query systemd's journal: service, boot and kernel logs

Intermediatelogssystemdtroubleshooting

journalctl reads the systemd journal, which collects stdout and stderr of every service, syslog messages and kernel messages in one indexed store. You filter by unit, time, priority or boot, follow logs live, and manage how much disk the journal uses.

Syntaxjournalctl [OPTIONS] [MATCHES]...

Watch it run

Users report 502s. Look at nginx's recent log entries, then watch new ones while you reproduce the problem.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 2
journalctl -u nginx -n 5 --no-pager

-u shows the unit's own output plus systemd's messages about it (start, reload, exit). The format is timestamp, host, process[PID], message.

Anatomy of a command

$
commandjournalctl

The reader for the binary journal files in /var/log/journal (or /run/log/journal).

How it works

systemd-journald collects from several sources: the stdout and stderr streams of every service (systemd connects them to a socket at start), the syslog socket /dev/log, the native journal API (sd_journal_send) and the kernel ring buffer (/dev/kmsg). It stores each message as a structured entry with trusted fields it adds itself, such as _PID, _UID, _SYSTEMD_UNIT, _BOOT_ID and _HOSTNAME, plus MESSAGE and PRIORITY. Entries are written to indexed binary files in /var/log/journal/<machine-id>/ (persistent) or /run/log/journal (memory only, lost at reboot). journalctl reads those files directly and uses the field indexes, so -u nginx or _PID=1234 doesn't scan the whole log. Size is bounded by SystemMaxUse (default 10% of the filesystem, capped at 4G) in journald.conf. Files are rotated and old ones deleted automatically, and --vacuum-* deletes archived files on demand. Non-root users see only their own entries unless they're in the adm or systemd-journal group.

Options you'll actually use

-u UNITEntries for one unit (repeat for several)
-fFollow new entries
-n NLast N entries (default 10 with -n alone)
--since / --untilTime window: '09:00', 'yesterday', '-1h'
-p PRIOPriority filter: emerg alert crit err warning notice info debug
-b [N]This boot, or -b -1 for the previous one
-kKernel messages only (like dmesg, but kept across boots)
-xAdd explanation text from the message catalog
-eJump to the end in the pager
-g PATTERNGrep MESSAGE with a regular expression
-o short-iso|json|catOutput format
--no-pagerPrint straight to stdout (scripts, pipes)
--list-bootsBoots recorded in the journal
--disk-usage / --vacuum-size=Journal size / delete archived files down to a size

Common mistakes

journalctl | grep nginx

It decodes the entire journal and pipes it through grep, which is slow, and it also matches unrelated lines that mention nginx.

journalctl -u nginx --since today
Expecting yesterday's boot logs with journalctl -b -1 on a fresh Ubuntu cloud image

If /var/log/journal doesn't exist, the journal is volatile (in /run) and lost at reboot.

sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald # or Storage=persistent
sudo rm /var/log/journal/*/*.journal to free space

Deleting files journald has open leaves the space in use and can corrupt the active file.

sudo journalctl --vacuum-size=500M (or --vacuum-time=7d)
Using -f in a script or cron job

-f never exits, so the script hangs forever.

journalctl -u app --since -5m --no-pager

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    The sandbox is a container without journald. Write the command that shows the last 50 lines of the ssh service's log, without a pager.

  2. 2

    Write the command that shows only errors (and worse) from the current boot.

  3. 3

    Write the command that follows nginx's log, starting with entries from the last 10 minutes.

  4. 4

    The sandbox has classic log files instead. Find the ERROR lines from /var/log/app/app.log, as you would with journalctl -p err for a service.

Related commands

systemctlStart, stop, enable and inspect systemd servicesdmesgRead the kernel's message logtailShow the last lines of a file and follow it livegrepSearch text for lines that match a pattern
PrevioussystemctlNextcrontab
Linux / Packages & Services / journalctl

journalctl

Query systemd's journal: service, boot and kernel logs

Intermediatelogssystemdtroubleshooting

journalctl reads the systemd journal, which collects stdout and stderr of every service, syslog messages and kernel messages in one indexed store. You filter by unit, time, priority or boot, follow logs live, and manage how much disk the journal uses.

Syntaxjournalctl [OPTIONS] [MATCHES]...

Watch it run

Users report 502s. Look at nginx's recent log entries, then watch new ones while you reproduce the problem.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 2
journalctl -u nginx -n 5 --no-pager

-u shows the unit's own output plus systemd's messages about it (start, reload, exit). The format is timestamp, host, process[PID], message.

Anatomy of a command

$
commandjournalctl

The reader for the binary journal files in /var/log/journal (or /run/log/journal).

How it works

systemd-journald collects from several sources: the stdout and stderr streams of every service (systemd connects them to a socket at start), the syslog socket /dev/log, the native journal API (sd_journal_send) and the kernel ring buffer (/dev/kmsg). It stores each message as a structured entry with trusted fields it adds itself, such as _PID, _UID, _SYSTEMD_UNIT, _BOOT_ID and _HOSTNAME, plus MESSAGE and PRIORITY. Entries are written to indexed binary files in /var/log/journal/<machine-id>/ (persistent) or /run/log/journal (memory only, lost at reboot). journalctl reads those files directly and uses the field indexes, so -u nginx or _PID=1234 doesn't scan the whole log. Size is bounded by SystemMaxUse (default 10% of the filesystem, capped at 4G) in journald.conf. Files are rotated and old ones deleted automatically, and --vacuum-* deletes archived files on demand. Non-root users see only their own entries unless they're in the adm or systemd-journal group.

Options you'll actually use

-u UNITEntries for one unit (repeat for several)
-fFollow new entries
-n NLast N entries (default 10 with -n alone)
--since / --untilTime window: '09:00', 'yesterday', '-1h'
-p PRIOPriority filter: emerg alert crit err warning notice info debug
-b [N]This boot, or -b -1 for the previous one
-kKernel messages only (like dmesg, but kept across boots)
-xAdd explanation text from the message catalog
-eJump to the end in the pager
-g PATTERNGrep MESSAGE with a regular expression
-o short-iso|json|catOutput format
--no-pagerPrint straight to stdout (scripts, pipes)
--list-bootsBoots recorded in the journal
--disk-usage / --vacuum-size=Journal size / delete archived files down to a size

Common mistakes

journalctl | grep nginx

It decodes the entire journal and pipes it through grep, which is slow, and it also matches unrelated lines that mention nginx.

journalctl -u nginx --since today
Expecting yesterday's boot logs with journalctl -b -1 on a fresh Ubuntu cloud image

If /var/log/journal doesn't exist, the journal is volatile (in /run) and lost at reboot.

sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald # or Storage=persistent
sudo rm /var/log/journal/*/*.journal to free space

Deleting files journald has open leaves the space in use and can corrupt the active file.

sudo journalctl --vacuum-size=500M (or --vacuum-time=7d)
Using -f in a script or cron job

-f never exits, so the script hangs forever.

journalctl -u app --since -5m --no-pager

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    The sandbox is a container without journald. Write the command that shows the last 50 lines of the ssh service's log, without a pager.

  2. 2

    Write the command that shows only errors (and worse) from the current boot.

  3. 3

    Write the command that follows nginx's log, starting with entries from the last 10 minutes.

  4. 4

    The sandbox has classic log files instead. Find the ERROR lines from /var/log/app/app.log, as you would with journalctl -p err for a service.

Related commands

systemctlStart, stop, enable and inspect systemd servicesdmesgRead the kernel's message logtailShow the last lines of a file and follow it livegrepSearch text for lines that match a pattern
PrevioussystemctlNextcrontab