Tracing Scripts

Writing Performance Tracing Scripts

Performance tracing with bpftrace involves crafting scripts that leverage eBPF programs to collect low-overhead metrics from the Linux kernel. This section demonstrates how to write scripts for measuring latency, CPU usage, and I/O operations using bpftrace. These examples focus on standard tracepoints and kprobes, ensuring compatibility across modern Linux kernels.

---

Latency Measurement

To measure latency for a specific function or system call, use kprobe (for entry) and kretprobe (for return). For example, trace the latency of the sys_read system call:

bpftrace -e '
kprobe:sys_read { $start = ns; }
kretprobe:sys_read { 
    printf("Read latency: %d ns\n", ns - $start); 
    trace("Read latency: %d ns", ns - $start);
}
'

Explanation:
- kprobe:sys_read captures the start time of the sys_read call.
- kretprobe:sys_read calculates the duration between entry and return.
- printf outputs the latency in nanoseconds.

For user-space functions, replace sys_read with the appropriate function name (e.g., do_read).

---

CPU Usage Tracking

Track CPU usage by aggregating time spent on each CPU core using the cpu:cpu_clock tracepoint. Note that cpu:cpu_clock returns clock cycles, not time units. To convert cycles to nanoseconds, use the CPU frequency (e.g., from /proc/cpuinfo or sysfs):

bpftrace -e '
BEGIN {
    $cpu_freq = 3500000000; // Example: 3.5 GHz
    timer:sleep(1s, @print_cpu_usage);
}

tracepoint:cpu:cpu_clock {
    @cpu_time[args->cpu] += args->clock;
}

@print_cpu_usage {
    printf("CPU %d time: %d cycles (%.2f ns)\n", 
           args->cpu, @cpu_time[args->cpu], 
           @cpu_time[args->cpu] * 1e9 / $cpu_freq);
    timer:sleep(1s, @print_cpu_usage);
}
'

Explanation:
- cpu:cpu_clock provides the CPU clock cycles for each core.
- The script uses a hash map @cpu_time to track per-CPU cycles.
- To convert cycles to nanoseconds, use the formula:
nanoseconds = (cycles * 1e9) / cpu_frequency.
Example: A 3.5 GHz CPU has 1 cycle = 0.2857 ns.

For process-level CPU usage, use kprobe/kretprobe with pid tracking instead of relying on cpu:cpu_clock:

bpftrace -e '
kprobe:sys_read { $start[pid] = ns; }
kretprobe:sys_read { 
    printf("PID %d Read latency: %d ns\n", pid, ns - $start[pid]); 
}
'

---

I/O Operation Monitoring

Monitor I/O operations by tracing relevant syscalls like read and write. Add metrics for bytes transferred and file descriptors:

bpftrace -e '
tracepoint:syscall:sys_enter_read { 
    $reads[pid]++; 
    $read_bytes[pid] += args->count; 
    printf("Read: %d bytes (fd: %d)\n", $read_bytes[pid], args->fd); 
}
tracepoint:syscall:sys_enter_write { 
    $writes[pid]++; 
    $write_bytes[pid] += args->count; 
    printf("Write: %d bytes (fd: %d)\n", $write_bytes[pid], args->fd); 
}
'

Explanation:
- sys_enter_read and sys_enter_write capture I/O syscall entries.
- The script counts and prints the number of read/write operations per process, along with bytes transferred and file descriptors.

For detailed timing, combine with latency tracking:

bpftrace -e '
kprobe:sys_read { $start = ns; }
kretprobe:sys_read { 
    printf("Read latency: %d ns\n", ns - $start); 
}
'

---

Key takeaways

• Use kprobe/kretprobe for precise latency measurements.
• Track CPU usage with cpu:cpu_clock (converted to nanoseconds) or sched:sched_switch.
• Monitor I/O operations via syscall tracepoints like sys_enter_read, including bytes and file descriptors.
• Always run scripts with sudo for kernel tracing privileges.
• Leverage printf and aggregation for actionable metrics.