Skip to content

eBPF Tracing

Linux systems generate vast amounts of performance data, and eBPF (Extended Berkeley Packet Filter) provides a low-overhead, flexible mechanism to monitor and analyze this data in real time. By leveraging eBPF programs, sysadmins can trace system calls, track I/O patterns, and analyze memory allocation without modifying kernel code. This section demonstrates practical use cases and tools for performance monitoring with eBPF.

Tracing System Calls with eBPF

System calls are critical for understanding application behavior and resource usage. eBPF programs can trace syscall entry and exit events, enabling detailed analysis of process activity.

Example: Counting open() calls

bpftrace -e 'tracepoint/syscalls/sys_enter_open { printf("PID %d opened %s\n", pid, str(retval)); }'
This program logs every open() syscall, including the file path and process ID. For production use, filter by specific paths or processes using if conditions.

Example: Aggregating syscall counts

perf stat -e syscalls:sys_enter_open,syscalls:sys_enter_read,syscalls:sys_enter_write -a
perf collects syscall statistics across the system, showing metrics like total calls, time spent, and context switches. Combine this with perf report to identify bottlenecks.

Key considerations:
- Use bpftrace for lightweight, inline tracing.
- perf is better for long-term, high-resolution data collection.
- Avoid tracing all syscalls; focus on critical operations like read, write, and mmap.

Tracking I/O Patterns with eBPF

I/O operations (disk, network, and file) are major performance bottlenecks. eBPF programs can capture I/O metadata, such as bytes transferred, timestamps, and file paths.

Example: Monitoring disk I/O

bpftrace -e 'tracepoint/block/block_rq_complete { printf("Disk I/O %d bytes\n", args->bytes); }'
This program logs completed block I/O operations, showing the number of bytes transferred. For network I/O, use tracepoint/net events like tcp_sendmsg or tcp_recvmsg.

Example: Analyzing I/O latency

perf record -e block:block_rq_start,block:block_rq_end -a
perf report --sort comm,dso,symbol
perf measures I/O latency by timing the start and end of block requests. Use --sort to prioritize high-latency operations.

Key considerations:
- Combine I/O tracing with process IDs to correlate activity with specific applications.
- Use tracepoint/net for network I/O, but note that it requires kernel support (e.g., CONFIG_NETFILTER_XT_TARGET_TRACE).
- For real-time monitoring, prioritize bpftrace over perf due to lower overhead.

Analyzing Memory Allocation with eBPF

Memory allocation patterns can reveal leaks, fragmentation, or inefficient usage. eBPF programs can trace kmalloc/kfree calls and track slab allocations.

Example: Detecting memory leaks

bpftrace -e 'tracepoint/sched/sched_process_fork { printf("PID %d allocated %d bytes\n", pid, args->stack); }'
This example logs memory allocations during process forks. For kernel memory, use tracepoint/slab events:
bpftrace -e 'tracepoint/slab/slab_free { printf("Freed %d bytes from %s\n", args->size, args->name); }'
Example: Monitoring memory usage with perf
perf record -e mem:mem_alloc,mem:mem_free -a
perf report --sort comm,dso,symbol
perf tracks memory allocation and deallocation events, helping identify leaks or excessive allocations.

Key considerations:
- Use memleak from BCC for advanced memory analysis (e.g., tracking malloc/free in user-space).
- Kernel memory tracing requires CONFIG_SLAB and CONFIG_BPF_SYSCALL enabled.
- Combine memory analysis with CPU and I/O metrics for holistic performance profiling.

Key takeaways

  • eBPF enables low-overhead, real-time monitoring of syscalls, I/O, and memory without kernel modifications.
  • Tracing syscalls helps identify application behavior and resource bottlenecks.
  • I/O analysis with eBPF or perf reveals disk/network latency and throughput issues.
  • Memory tracking detects leaks and allocation patterns, critical for optimizing application performance.
  • Always balance granularity with performance overhead, and validate results with multiple tools.