CrackOps
Interview PrepTracksLinuxAWS LabSoonInterview RoomSoon
Sign inStart free
CrackOps

Interview prep for the people who keep production up.

[email protected]
ProductInterview PrepTracksDashboardInterview Room
RolesDevOpsSREPlatformCloud
CompanyAboutContactPrivacyTerms
© CrackOps
Linux / Docker / docker stats

docker stats

Live CPU, memory, network and I/O per container

Beginnermonitoringperformancecgroups

docker stats shows a live table of resource use for each running container: CPU %, memory used against its limit, network and block I/O, and process count. It's top for containers, and the quickest way to find which one is eating the host.

Syntaxdocker stats [OPTIONS] [CONTAINER...]

Watch it run

You want a quick picture of what every container on this host is using right now.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 2
docker stats --no-stream

One row per running container. Only api has a memory limit (512MiB); the others show the host's total RAM as their limit, meaning they could use all of it. NET I/O is received / sent, BLOCK I/O is read / written.

Anatomy of a command

$
commanddocker stats

Stream resource usage for running containers, refreshing about once a second.

How it works

For each container the CLI opens GET /containers/<id>/stats, and dockerd samples the container's cgroup files about once a second. CPU % is the change in the cgroup's CPU time divided by the change in total system CPU time, times the number of CPUs, so 200% means two full cores. MEM USAGE is memory.current minus inactive file cache (page cache the kernel can drop), and LIMIT is memory.max, or the host's RAM if there's no -m. NET I/O comes from the interfaces in the container's network namespace, BLOCK I/O from io.stat, and PIDS from pids.current, counting threads too. Because CPU needs two samples, --no-stream takes about two seconds. The numbers are the same ones the kernel uses to OOM-kill and throttle the container.

Options you'll actually use

--no-streamOne snapshot, then exit
-a, --allInclude stopped containers (shown with zeros)
--format TEMPLATEColumns: .Name .CPUPerc .MemUsage .MemPerc .NetIO .BlockIO .PIDs
--no-truncDon't truncate container IDs
docker top CONTAINERThe processes inside one container (companion command)
docker system dfDisk use by images, containers, volumes and cache

Common mistakes

Reading 150% CPU as 'over the limit'

CPU % is relative to one core; 150% means one and a half cores busy. It's only a problem relative to --cpus or the host's core count.

Compare with --cpus (docker inspect -f '{{.HostConfig.NanoCpus}}') and nproc
Comparing docker stats memory with free or top inside the container

Inside, tools read /proc/meminfo, which shows the host's memory, not the cgroup's limit or usage.

Trust docker stats, or read /sys/fs/cgroup/memory.current inside the container
Running containers without -m and relying on docker stats to notice problems

By the time you look, the host may already be in a global OOM state.

Set -m (and --cpus) on every container; use stats to tune them

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Read the api container's memory limit from cgroup v2 inside it, then read your own shell's memory use and limit on this machine.

  2. 2

    Write a docker command that prints, without headers, the name and CPU % of every running container, highest CPU first.

  3. 3

    Read this row and say what's wrong: 'worker 0.40% 1.92GiB / 15.62GiB 12.29% ... PIDS 4096'.

Related commands

docker inspectShow the full configuration and state of Docker objectstopWatch CPU, memory and processes livefreeShow used, free and available memorykubectl topShow CPU and memory use of nodes and podsdocker psList containers and their state
Previousdocker inspectNextdocker network
Linux / Docker / docker stats

docker stats

Live CPU, memory, network and I/O per container

Beginnermonitoringperformancecgroups

docker stats shows a live table of resource use for each running container: CPU %, memory used against its limit, network and block I/O, and process count. It's top for containers, and the quickest way to find which one is eating the host.

Syntaxdocker stats [OPTIONS] [CONTAINER...]

Watch it run

You want a quick picture of what every container on this host is using right now.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 2
docker stats --no-stream

One row per running container. Only api has a memory limit (512MiB); the others show the host's total RAM as their limit, meaning they could use all of it. NET I/O is received / sent, BLOCK I/O is read / written.

Anatomy of a command

$
commanddocker stats

Stream resource usage for running containers, refreshing about once a second.

How it works

For each container the CLI opens GET /containers/<id>/stats, and dockerd samples the container's cgroup files about once a second. CPU % is the change in the cgroup's CPU time divided by the change in total system CPU time, times the number of CPUs, so 200% means two full cores. MEM USAGE is memory.current minus inactive file cache (page cache the kernel can drop), and LIMIT is memory.max, or the host's RAM if there's no -m. NET I/O comes from the interfaces in the container's network namespace, BLOCK I/O from io.stat, and PIDS from pids.current, counting threads too. Because CPU needs two samples, --no-stream takes about two seconds. The numbers are the same ones the kernel uses to OOM-kill and throttle the container.

Options you'll actually use

--no-streamOne snapshot, then exit
-a, --allInclude stopped containers (shown with zeros)
--format TEMPLATEColumns: .Name .CPUPerc .MemUsage .MemPerc .NetIO .BlockIO .PIDs
--no-truncDon't truncate container IDs
docker top CONTAINERThe processes inside one container (companion command)
docker system dfDisk use by images, containers, volumes and cache

Common mistakes

Reading 150% CPU as 'over the limit'

CPU % is relative to one core; 150% means one and a half cores busy. It's only a problem relative to --cpus or the host's core count.

Compare with --cpus (docker inspect -f '{{.HostConfig.NanoCpus}}') and nproc
Comparing docker stats memory with free or top inside the container

Inside, tools read /proc/meminfo, which shows the host's memory, not the cgroup's limit or usage.

Trust docker stats, or read /sys/fs/cgroup/memory.current inside the container
Running containers without -m and relying on docker stats to notice problems

By the time you look, the host may already be in a global OOM state.

Set -m (and --cpus) on every container; use stats to tune them

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Read the api container's memory limit from cgroup v2 inside it, then read your own shell's memory use and limit on this machine.

  2. 2

    Write a docker command that prints, without headers, the name and CPU % of every running container, highest CPU first.

  3. 3

    Read this row and say what's wrong: 'worker 0.40% 1.92GiB / 15.62GiB 12.29% ... PIDS 4096'.

Related commands

docker inspectShow the full configuration and state of Docker objectstopWatch CPU, memory and processes livefreeShow used, free and available memorykubectl topShow CPU and memory use of nodes and podsdocker psList containers and their state
Previousdocker inspectNextdocker network