Skip to content

Profiling Data

Interpreting perf and SystemTap Output

perf Data Analysis

The perf tool provides detailed insights into CPU usage, function call stacks, and hardware events. Use perf report to analyze recorded data:

perf report --sort comm,dso
This sorts results by process name (comm) and dynamic shared object (dso). Look for:
- CPU usage: High %CPU values indicate bottlenecks.
- Function calls: Stack traces (SYMBOLS column) reveal hotspots.
- DSO (Dynamic Shared Object): Identify kernel modules or user-space libraries contributing to overhead.

For visualizing call stacks, use perf script with a flamegraph generator:

perf script > trace.out && FlameGraph/FlameGraph.pl --color=always trace.out > flamegraph.svg
Flamegraphs stack traces vertically, making it easy to spot functions consuming the most cycles.

SystemTap Output Interpretation

SystemTap scripts (stap) generate logs that show event triggers and metric aggregations. Example output:

# stap -e 'probe kernel.function("sys_read") { printf("Read %d bytes\n", args->count) }'
Read 4096 bytes
Read 2048 bytes
- Event triggers: probe kernel.function(...) indicates when a function is called.
- Metrics: Use printf or histogram to aggregate data (e.g., I/O sizes, latency).
- Logs: The --log flag adds debug output for complex scripts.

Identifying Hotspots

Use perf top to monitor real-time function activity:

perf top -p <PID>
Sort by CPU usage with --sort comm,dso. For deeper analysis:
perf record -aR -g sleep 10
perf report --sort comm,dso
This records all processes for 10 seconds, then reports aggregated data.

Correlating Metrics with System Behavior

Combine profiling data with system logs and monitoring tools:
1. Check system logs: Use dmesg or journalctl to correlate kernel messages with high CPU usage.
2. Monitor I/O: Pair perf data with iostat -m 1 to identify disk bottlenecks.
3. Track memory: Use vmstat or perf's --branch-stack to analyze memory-related slowdowns.

Key takeaways

  • Use perf report and flamegraphs to visualize CPU and function-level bottlenecks.
  • SystemTap scripts enable granular event tracking, ideal for custom metric aggregation.
  • Sort perf data by comm,dso to isolate high-impact processes or modules.
  • Correlate profiling data with system logs and tools like iostat to diagnose root causes.
  • Leverage perf top for real-time monitoring and perf record for detailed post-mortem analysis.