Skip to content

perf Suite

The perf tool suite is a powerful set of command-line utilities for profiling Linux systems, enabling detailed analysis of CPU usage, function call stacks, and kernel behavior. It leverages the Linux kernel's Performance Events subsystem to collect data from hardware and software events, making it indispensable for diagnosing performance bottlenecks and optimizing system behavior. This section covers how to use perf to profile CPU usage, track function calls, and analyze kernel stack traces.

Profiling CPU Usage

To profile CPU usage, perf captures samples of CPU activity over time, revealing which processes or functions consume the most resources.

Example: Recording CPU usage

sudo perf record -g sleep 10
- -g: Enables call graph recording (for function call tracking).
- sleep 10: Runs a dummy workload for 10 seconds.

Analyzing results

sudo perf report
- Displays a flame graph-like view of sampled functions, sorted by CPU time.
- Use --sort comm,dso to sort by process name or shared object (library).

Statistical overview

sudo perf stat -e cpu-clock,context-switches,cpu-migrations sleep 5
- Measures events like CPU clock cycles, context switches, and migrations.

Tracking Function Calls

The -g option enables call graph recording, allowing you to trace how functions invoke each other.

Example: Recording function call stacks

sudo perf record -g --call-graph fp
- --call-graph fp: Uses frame pointers for call stack recording (requires kernel support).

Analyzing call graphs

sudo perf report --call-graph fp
- Displays a hierarchical view of function invocations.
- Use --sort comm,dt to sort by process and duration.

Filtering by process

sudo perf record -g -p <PID> sleep 5
- Replaces <PID> with a specific process ID to focus on its behavior.

Analyzing Kernel Stack Traces

To debug kernel-level performance issues, perf can capture stack traces of kernel functions.

Example: Recording kernel events

sudo perf record -g -k /usr/lib/debug/boot/vmlinux-unsigned
- -k: Specifies the kernel debug symbol file (ensure it matches your kernel version).

Analyzing kernel stack traces

sudo perf report --kallsyms
- Filters output to show kernel symbols and stack traces.
- Look for patterns in kernel functions like do_syscall_64 or schedule.

Troubleshooting tips
- If no kernel symbols are found, rebuild your kernel with CONFIG_DEBUG_INFO enabled.
- Use perf list to check available events and ensure perf is properly installed.

Key takeaways

  • Use perf record -g to capture detailed CPU and function call data.
  • Analyze results with perf report to identify performance bottlenecks.
  • Kernel stack traces require debug symbols and proper configuration.
  • Combine perf stat for statistical insights and perf record for granular tracing.
  • Always verify kernel version and debug symbol compatibility for kernel analysis.