numactlControl NUMA CPU and memory placement
On servers with more than one CPU socket, each socket has its own 'local' memory and reaching the other socket's memory is slower. numactl shows that layout and runs a program with a chosen CPU and memory policy: pinned to one node, or interleaved across all of them.
numactl [--hardware | --show] | numactl [--cpunodebind=N] [--membind=N | --interleave=all | --preferred=N] [--] COMMAND [ARG]...Before tuning anything you want to know whether NUMA even matters on this machine.
numactlSets a NUMA policy for itself, then execs the command, which inherits it.
On a multi-socket machine the memory controllers live in the CPUs, so each socket (a NUMA node) has fast local RAM and slower remote RAM reached over the interconnect. The kernel publishes the layout in /sys/devices/system/node: CPUs per node, memory per node and a distance table (10 = local; 20-32 = remote, roughly 1.5-2x the latency). By default Linux allocates a page on the node of the CPU that first touches it. numactl changes that for one program: it calls sched_setaffinity for --cpunodebind/--physcpubind and set_mempolicy for --membind (only these nodes), --preferred (try this node first) or --interleave (round-robin pages across nodes), then execs the command; children inherit both. numastat reads per-node counters (numa_hit, numa_miss, other_node) and per-process placement from /proc/PID/numa_maps. Most cloud VMs below the largest sizes expose a single node, where numactl changes nothing.
--hardware, -HNodes, their CPUs, memory size and free memory, distances--show, -sThe NUMA policy and CPU binding of the current process--cpunodebind=NRun on the CPUs of node(s) N (e.g. 0 or 0,1)--physcpubind=LISTRun on specific CPUs, e.g. 0-7--membind=NAllocate only from node(s) N; fail (or OOM) if they're full--preferred=NPrefer node N, fall back to others when it's full--interleave=allSpread pages round-robin across nodes (databases with huge caches)--localallocAlways allocate on the node the thread is running onnumastat [-p PID|NAME]Per-node hit/miss counters; per-process memory by nodeAdding numactl tuning on a cloud VM with a single NUMA nodeWith one node there is no remote memory; the policy has no effect and only makes the start command harder to read.
Check numactl --hardware or lscpu firstUsing --membind for a process whose memory can exceed one nodemembind is strict: when that node is full the process is OOM-killed even if other nodes are empty.
--preferred=N to prefer a node but allow spill, or --interleave=all for big shared cachesPinning CPUs with --physcpubind but not the memoryThreads run on node 0 while memory first touched elsewhere stays remote, so you get the restriction without the benefit.
Pair --cpunodebind=N with --membind=N or --localallocTry these on a Linux machine of your own. Each task has a hint and a solution.
Find how many NUMA nodes this machine has and how much memory each one has free.
Show the NUMA policy and CPU binding of your current shell.
Run 'numactl --show' under an interleave policy across all nodes and read the policy line.
numactlControl NUMA CPU and memory placement
On servers with more than one CPU socket, each socket has its own 'local' memory and reaching the other socket's memory is slower. numactl shows that layout and runs a program with a chosen CPU and memory policy: pinned to one node, or interleaved across all of them.
numactl [--hardware | --show] | numactl [--cpunodebind=N] [--membind=N | --interleave=all | --preferred=N] [--] COMMAND [ARG]...Before tuning anything you want to know whether NUMA even matters on this machine.
numactlSets a NUMA policy for itself, then execs the command, which inherits it.
On a multi-socket machine the memory controllers live in the CPUs, so each socket (a NUMA node) has fast local RAM and slower remote RAM reached over the interconnect. The kernel publishes the layout in /sys/devices/system/node: CPUs per node, memory per node and a distance table (10 = local; 20-32 = remote, roughly 1.5-2x the latency). By default Linux allocates a page on the node of the CPU that first touches it. numactl changes that for one program: it calls sched_setaffinity for --cpunodebind/--physcpubind and set_mempolicy for --membind (only these nodes), --preferred (try this node first) or --interleave (round-robin pages across nodes), then execs the command; children inherit both. numastat reads per-node counters (numa_hit, numa_miss, other_node) and per-process placement from /proc/PID/numa_maps. Most cloud VMs below the largest sizes expose a single node, where numactl changes nothing.
--hardware, -HNodes, their CPUs, memory size and free memory, distances--show, -sThe NUMA policy and CPU binding of the current process--cpunodebind=NRun on the CPUs of node(s) N (e.g. 0 or 0,1)--physcpubind=LISTRun on specific CPUs, e.g. 0-7--membind=NAllocate only from node(s) N; fail (or OOM) if they're full--preferred=NPrefer node N, fall back to others when it's full--interleave=allSpread pages round-robin across nodes (databases with huge caches)--localallocAlways allocate on the node the thread is running onnumastat [-p PID|NAME]Per-node hit/miss counters; per-process memory by nodeAdding numactl tuning on a cloud VM with a single NUMA nodeWith one node there is no remote memory; the policy has no effect and only makes the start command harder to read.
Check numactl --hardware or lscpu firstUsing --membind for a process whose memory can exceed one nodemembind is strict: when that node is full the process is OOM-killed even if other nodes are empty.
--preferred=N to prefer a node but allow spill, or --interleave=all for big shared cachesPinning CPUs with --physcpubind but not the memoryThreads run on node 0 while memory first touched elsewhere stays remote, so you get the restriction without the benefit.
Pair --cpunodebind=N with --membind=N or --localallocTry these on a Linux machine of your own. Each task has a hint and a solution.
Find how many NUMA nodes this machine has and how much memory each one has free.
Show the NUMA policy and CPU binding of your current shell.
Run 'numactl --show' under an interleave policy across all nodes and read the policy line.