Profiling CPU/Memory

Linux sysadmins often need to identify CPU and memory bottlenecks to optimize system performance. eBPF (extended Berkeley Packet Filter) and the BCC (BPF Compiler Collection) toolkit provide powerful, low-overhead tools for profiling these resources. This section demonstrates how to use BCC-based tools like bpftrace, flamegraph, and perf to analyze CPU and memory usage, uncovering performance issues in real time.

CPU Profiling with BCC¶

Overview¶

CPU profiling helps identify which processes, functions, or system calls consume the most CPU time. BCC integrates with the Linux perf tool and bpftrace to capture detailed CPU usage metrics, enabling the creation of flamegraphs that visualize stack traces over time.

Tools¶

perf_events: A kernel-level performance monitoring framework for collecting CPU events. BCC tools like bpftrace leverage this framework internally for eBPF-based profiling, but perf_events itself is a separate kernel component from BCC.
bpftrace: A high-level scripting tool for writing eBPF programs.
flamegraph: A script to generate visual flamegraphs from perf data.

Example: CPU Flamegraph¶

Collect CPU data using perf:
```
sudo perf record -g -- sleep 10
```
This records CPU events for 10 seconds.
Generate a flamegraph:
```
sudo perf script | ./flamegraph --color=always > cpu_flame.svg
```
Replace ./flamegraph with the path to the FlameGraph script (downloaded from https://github.com/brendangregg/FlameGraph).
Analyze the SVG file to see where CPU time is spent.

Example: bpftrace CPU Tracing¶

Use bpftrace to trace CPU usage per process:

bpftrace -e 'tracepoint:sched:sched_switch { printf("CPU %d: %s -> %s\n", cpu, str(retval), str(args->next->comm)); }'

This outputs context switches, useful for detecting CPU contention or scheduling issues.

Memory Profiling with BCC¶

Overview¶

Memory profiling identifies allocation patterns, leaks, or contention issues. BCC tools like bpftrace and mem can track memory events, such as allocations, page faults, and deallocations, to pinpoint inefficiencies.

Tools¶

bpftrace: For custom memory event tracing.
mem: A BCC tool for monitoring memory usage (e.g., tracks per-process memory statistics, page faults, and allocation patterns).
perf: For tracking memory-related system calls.

Example: Memory Allocation Tracking¶

Trace memory allocations with bpftrace:

bpftrace -e 'tracepoint:kmem:kmalloc { printf("Allocated %d bytes at %p\n", args->size, args->ptr); }'

This logs allocation sizes and addresses, helping identify memory-heavy operations. Note that kmem:kmalloc requires a kernel with CONFIG_KMEM_ACCOUNTING enabled.

Example: Page Fault Monitoring¶

Detect page faults (indicative of memory pressure):

bpftrace -e 'tracepoint:pagefault:pagefault { printf("Page fault on %p\n", args->address); }'

This helps identify applications causing frequent memory paging.

Key takeaways¶

CPU profiling with BCC combines perf and flamegraph for visualizing stack traces and identifying hotspots. perf_events is a kernel framework used by BCC tools like bpftrace for low-level event collection.
Memory profiling leverages bpftrace and mem to track allocations, page faults, and process-specific usage.
These tools operate with minimal overhead, making them ideal for production environments. Always validate findings with additional metrics (e.g., top, vmstat) for accuracy.