perfProfile where CPU time goes, kernel and user space
perf is the Linux kernel's own profiler. It counts hardware and software events (perf stat), shows the hottest functions live (perf top) and records stack samples you can turn into reports and flame graphs (perf record / report), with low enough overhead to use on production.
perf stat [-e EVENTS] CMD | perf top [-p PID] | perf record [-g] [-F HZ] [-p PID] [-- CMD] | perf reportYou want to profile a CPU-hungry service on an Ubuntu EC2 instance for the first time.
sudoProfiling other users' processes and the kernel needs root or CAP_PERFMON (see perf_event_paranoid).
perf talks to the kernel through perf_event_open(2). perf stat asks the kernel to count events: software ones (task-clock, context switches, page faults) and hardware ones from the CPU's performance monitoring unit (cycles, instructions, cache and branch misses). perf record instead programs an interrupt every N events or at a fixed frequency (-F 99). On each interrupt the kernel notes which instruction was running, the PID and, with -g, walks the call stack; samples go into a ring buffer shared with perf, which writes them to perf.data. perf report later maps addresses to function names using the binaries' symbol tables, so stripped binaries, JITs (Java, Node) and code built without frame pointers show [unknown] or broken stacks unless you add symbols, perf maps or --call-graph dwarf. Overhead at 99 Hz is tiny compared with strace. On VMs, hardware counters are often not exposed, so perf falls back to the cpu-clock timer.
The kernel interrupts the CPU 99 times a second, notes what was running and its call stack, and perf turns those samples into a ranked report.
perf record calls perf_event_open for PID 4211: 'interrupt me 99 times a second on every CPU this process runs on, and capture the call stack'.
perf stat CMDCount events while CMD runs: task-clock, context switches, IPCperf topLive view of the hottest functions, system-wide or with -p PIDperf record -gSample with call stacks into perf.dataperf reportBrowse perf.data; --stdio for plain text, --no-children for self time-F HZSampling frequency per CPU (99 is a common, cheap choice)-p PID / -aOne process / every CPU, system-wide-e EVENTPick events: cycles, cache-misses, sched:sched_switch... (perf list)--call-graph dwarfUnwind stacks with debug info when frame pointers are missingperf tracestrace-like syscall tracing with far lower overheadperf scriptDump raw samples, e.g. to feed FlameGraph's stackcollapse-perf.plLowering kernel.perf_event_paranoid to -1 on every server so perf 'just works'It lets any local user profile any process and the kernel, which leaks information. It's a security setting.
Use sudo perf, or grant CAP_PERFMON to a specific binary or userProfiling a Java or Node service and getting mostly [unknown] or hex addressesJIT-compiled code has no symbols on disk; perf can't name it.
Node: --perf-basic-prof; Java: -XX:+PreserveFramePointer plus perf-map-agent or async-profilerperf record -g -a for 10 minutes at the default frequency on a 64-core hostThat's millions of samples: gigabytes of perf.data and a slow report, for no extra insight.
-F 99 for 30-60 seconds is enough to find a hot pathExpecting perf to show why a process is slow while it's sleeping on I/OOn-CPU sampling only sees code that is running. Time spent blocked doesn't produce samples.
For off-CPU time use strace -T, perf trace, or offcputime from bccTry these on a Linux machine of your own. Each task has a hint and a solution.
Count the cost of compressing a file: run 'gzip -k -f' on a file in ~/data under perf stat. The practice sandbox can't run perf (it needs the host kernel's perf events and a matching linux-tools package); write the commands and run them on a VM.
Record 10 seconds of system-wide samples with call stacks, then show the top 10 symbols as plain text. The practice sandbox can't run perf (it needs the host kernel's perf events and a matching linux-tools package); write the commands and run them on a VM.
Check how restrictive the kernel is about perf on this machine and explain the value.
perfProfile where CPU time goes, kernel and user space
perf is the Linux kernel's own profiler. It counts hardware and software events (perf stat), shows the hottest functions live (perf top) and records stack samples you can turn into reports and flame graphs (perf record / report), with low enough overhead to use on production.
perf stat [-e EVENTS] CMD | perf top [-p PID] | perf record [-g] [-F HZ] [-p PID] [-- CMD] | perf reportYou want to profile a CPU-hungry service on an Ubuntu EC2 instance for the first time.
sudoProfiling other users' processes and the kernel needs root or CAP_PERFMON (see perf_event_paranoid).
perf talks to the kernel through perf_event_open(2). perf stat asks the kernel to count events: software ones (task-clock, context switches, page faults) and hardware ones from the CPU's performance monitoring unit (cycles, instructions, cache and branch misses). perf record instead programs an interrupt every N events or at a fixed frequency (-F 99). On each interrupt the kernel notes which instruction was running, the PID and, with -g, walks the call stack; samples go into a ring buffer shared with perf, which writes them to perf.data. perf report later maps addresses to function names using the binaries' symbol tables, so stripped binaries, JITs (Java, Node) and code built without frame pointers show [unknown] or broken stacks unless you add symbols, perf maps or --call-graph dwarf. Overhead at 99 Hz is tiny compared with strace. On VMs, hardware counters are often not exposed, so perf falls back to the cpu-clock timer.
The kernel interrupts the CPU 99 times a second, notes what was running and its call stack, and perf turns those samples into a ranked report.
perf record calls perf_event_open for PID 4211: 'interrupt me 99 times a second on every CPU this process runs on, and capture the call stack'.
perf stat CMDCount events while CMD runs: task-clock, context switches, IPCperf topLive view of the hottest functions, system-wide or with -p PIDperf record -gSample with call stacks into perf.dataperf reportBrowse perf.data; --stdio for plain text, --no-children for self time-F HZSampling frequency per CPU (99 is a common, cheap choice)-p PID / -aOne process / every CPU, system-wide-e EVENTPick events: cycles, cache-misses, sched:sched_switch... (perf list)--call-graph dwarfUnwind stacks with debug info when frame pointers are missingperf tracestrace-like syscall tracing with far lower overheadperf scriptDump raw samples, e.g. to feed FlameGraph's stackcollapse-perf.plLowering kernel.perf_event_paranoid to -1 on every server so perf 'just works'It lets any local user profile any process and the kernel, which leaks information. It's a security setting.
Use sudo perf, or grant CAP_PERFMON to a specific binary or userProfiling a Java or Node service and getting mostly [unknown] or hex addressesJIT-compiled code has no symbols on disk; perf can't name it.
Node: --perf-basic-prof; Java: -XX:+PreserveFramePointer plus perf-map-agent or async-profilerperf record -g -a for 10 minutes at the default frequency on a 64-core hostThat's millions of samples: gigabytes of perf.data and a slow report, for no extra insight.
-F 99 for 30-60 seconds is enough to find a hot pathExpecting perf to show why a process is slow while it's sleeping on I/OOn-CPU sampling only sees code that is running. Time spent blocked doesn't produce samples.
For off-CPU time use strace -T, perf trace, or offcputime from bccTry these on a Linux machine of your own. Each task has a hint and a solution.
Count the cost of compressing a file: run 'gzip -k -f' on a file in ~/data under perf stat. The practice sandbox can't run perf (it needs the host kernel's perf events and a matching linux-tools package); write the commands and run them on a VM.
Record 10 seconds of system-wide samples with call stacks, then show the top 10 symbols as plain text. The practice sandbox can't run perf (it needs the host kernel's perf events and a matching linux-tools package); write the commands and run them on a VM.
Check how restrictive the kernel is about perf on this machine and explain the value.