CrackOps
Interview PrepTracksLinuxAWS LabSoonInterview RoomSoon
Sign inStart free
CrackOps

Interview prep for the people who keep production up.

[email protected]
ProductInterview PrepTracksDashboardInterview Room
RolesDevOpsSREPlatformCloud
CompanyAboutContactPrivacyTerms
© CrackOps
Linux / Performance Tuning / sar

sar

Collect and replay system activity history

Intermediateperformancemonitoringhistory

sar (System Activity Reporter, part of sysstat) reports CPU, memory, disk, network and load, either live every few seconds or from history files recorded every 10 minutes. It answers the question other tools can't: what was this box doing at 02:10 last night?

Syntaxsar [-u|-r|-q|-b|-d -p|-n DEV|-W|-A] [-f FILE] [-s HH:MM:SS] [-e HH:MM:SS] [INTERVAL [COUNT]]

Watch it run

Users report slowness right now and you want a quick per-second view of CPU, memory and load in one tool.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 3
sar -u 1 3

One row per second and an Average line at the end. The CPUs are almost never idle, and a fifth of the time they sit waiting for disk I/O (%iowait). Something is both computing and reading or writing heavily.

Anatomy of a command

$
commandsar

Reads activity data, live or from a daily binary file, and prints it as a time table.

How it works

sar is the report half of sysstat. The collector, sadc, reads kernel counters from /proc: /proc/stat for CPU, /proc/meminfo, /proc/loadavg, /proc/diskstats, /proc/net/dev, /proc/vmstat. Counters only ever increase, so a rate is the difference between two samples divided by the interval. In live mode (sar -u 1 3) sar starts sadc itself and prints each interval. For history, a systemd timer (sysstat-collect.timer, every 10 minutes) runs sa1, which appends a sample to a binary file per day: /var/log/sysstat/saDD on Ubuntu, /var/log/sa/saDD on RHEL and Amazon Linux. Another timer runs sa2 after midnight to write a text summary. Files are kept for HISTORY days (7 by default on Ubuntu). On Ubuntu collection is off until you set ENABLED="true" in /etc/default/sysstat. The 10-minute averages hide short spikes, so use live mode with a 1-second interval while an incident is happening.

Options you'll actually use

-uCPU: %user %nice %system %iowait %steal %idle (-P ALL per core)
-rMemory: free, available, used, buffers, cached, committed, dirty
-qRun queue length, process count, load averages, blocked tasks
-bTotal I/O transfers and blocks read/written per second
-d -pPer-disk tps, kB/s, await (ms) and %util, with device names
-n DEV / EDEVPer-interface packets and kB/s / interface errors and drops
-n TCP,ETCPTCP connections opened per second, retransmits, resets
-WSwapping: pages swapped in and out per second
-f FILERead a history file instead of live data
-s / -eStart / end time when reading history
INTERVAL COUNTLive mode: sample every INTERVAL seconds, COUNT times

Common mistakes

Installing sysstat on Ubuntu and assuming history is being recorded

Ubuntu ships it with ENABLED="false"; nothing is collected until you change it, so there's no data when you need it.

Set ENABLED="true" in /etc/default/sysstat and systemctl enable --now sysstat on every server image
Reading only the Average line of a history report

An average over an hour flattens a 20-minute spike into a mild bump.

Look at the individual rows, and narrow the window with -s and -e
Expecting sar history to catch a 30-second spike

History is sampled every 10 minutes and averaged; short bursts disappear.

During an incident use live mode: sar -u 1 60, or a monitoring agent with 10-60 s resolution
Looking for sa files in /var/log/sa on Ubuntu (or /var/log/sysstat on RHEL)

The path differs by distribution, so you conclude there's no data.

Ubuntu/Debian: /var/log/sysstat/saDD; RHEL, Amazon Linux: /var/log/sa/saDD

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Show CPU usage every 2 seconds, 5 times, then the average.

  2. 2

    Show memory and swap activity live, 3 samples one second apart.

  3. 3

    Check whether history collection is enabled on this machine, and list the history files if any.

Related commands

mpstatPer-CPU utilisation: find the hot corepidstatPer-process CPU, memory, disk and context switches over timevmstatReport CPU, memory, swap and I/O activity over timeiostatMeasure disk throughput, latency and utilisation
Previoustar --listed-incrementalNextmpstat
Linux / Performance Tuning / sar

sar

Collect and replay system activity history

Intermediateperformancemonitoringhistory

sar (System Activity Reporter, part of sysstat) reports CPU, memory, disk, network and load, either live every few seconds or from history files recorded every 10 minutes. It answers the question other tools can't: what was this box doing at 02:10 last night?

Syntaxsar [-u|-r|-q|-b|-d -p|-n DEV|-W|-A] [-f FILE] [-s HH:MM:SS] [-e HH:MM:SS] [INTERVAL [COUNT]]

Watch it run

Users report slowness right now and you want a quick per-second view of CPU, memory and load in one tool.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 3
sar -u 1 3

One row per second and an Average line at the end. The CPUs are almost never idle, and a fifth of the time they sit waiting for disk I/O (%iowait). Something is both computing and reading or writing heavily.

Anatomy of a command

$
commandsar

Reads activity data, live or from a daily binary file, and prints it as a time table.

How it works

sar is the report half of sysstat. The collector, sadc, reads kernel counters from /proc: /proc/stat for CPU, /proc/meminfo, /proc/loadavg, /proc/diskstats, /proc/net/dev, /proc/vmstat. Counters only ever increase, so a rate is the difference between two samples divided by the interval. In live mode (sar -u 1 3) sar starts sadc itself and prints each interval. For history, a systemd timer (sysstat-collect.timer, every 10 minutes) runs sa1, which appends a sample to a binary file per day: /var/log/sysstat/saDD on Ubuntu, /var/log/sa/saDD on RHEL and Amazon Linux. Another timer runs sa2 after midnight to write a text summary. Files are kept for HISTORY days (7 by default on Ubuntu). On Ubuntu collection is off until you set ENABLED="true" in /etc/default/sysstat. The 10-minute averages hide short spikes, so use live mode with a 1-second interval while an incident is happening.

Options you'll actually use

-uCPU: %user %nice %system %iowait %steal %idle (-P ALL per core)
-rMemory: free, available, used, buffers, cached, committed, dirty
-qRun queue length, process count, load averages, blocked tasks
-bTotal I/O transfers and blocks read/written per second
-d -pPer-disk tps, kB/s, await (ms) and %util, with device names
-n DEV / EDEVPer-interface packets and kB/s / interface errors and drops
-n TCP,ETCPTCP connections opened per second, retransmits, resets
-WSwapping: pages swapped in and out per second
-f FILERead a history file instead of live data
-s / -eStart / end time when reading history
INTERVAL COUNTLive mode: sample every INTERVAL seconds, COUNT times

Common mistakes

Installing sysstat on Ubuntu and assuming history is being recorded

Ubuntu ships it with ENABLED="false"; nothing is collected until you change it, so there's no data when you need it.

Set ENABLED="true" in /etc/default/sysstat and systemctl enable --now sysstat on every server image
Reading only the Average line of a history report

An average over an hour flattens a 20-minute spike into a mild bump.

Look at the individual rows, and narrow the window with -s and -e
Expecting sar history to catch a 30-second spike

History is sampled every 10 minutes and averaged; short bursts disappear.

During an incident use live mode: sar -u 1 60, or a monitoring agent with 10-60 s resolution
Looking for sa files in /var/log/sa on Ubuntu (or /var/log/sysstat on RHEL)

The path differs by distribution, so you conclude there's no data.

Ubuntu/Debian: /var/log/sysstat/saDD; RHEL, Amazon Linux: /var/log/sa/saDD

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Show CPU usage every 2 seconds, 5 times, then the average.

  2. 2

    Show memory and swap activity live, 3 samples one second apart.

  3. 3

    Check whether history collection is enabled on this machine, and list the history files if any.

Related commands

mpstatPer-CPU utilisation: find the hot corepidstatPer-process CPU, memory, disk and context switches over timevmstatReport CPU, memory, swap and I/O activity over timeiostatMeasure disk throughput, latency and utilisation
Previoustar --listed-incrementalNextmpstat