Probe Config

Advanced Probe Configuration

When instrumenting kernel or user-space code with eBPF probes, advanced configuration strategies are essential to ensure accuracy, performance, and avoid conflicts. This section covers probe placement, symbol resolution, and mitigation of instrumentation conflicts.

---

Probe Placement Strategies

1. Kernel vs. User-Space Probes
Kernel probes (e.g., kprobe, tracepoint) are ideal for tracing system calls, interrupt handling, or kernel functions. User-space probes (uprobe) are better suited for tracing application-specific functions (e.g., open(), read()).

Example:

# Trace sys_open in kernel space  
sudo bpftool prog load ./kprobe-open.o /sys/fs/bpf/kprobe-open  
sudo bpftool prog attach kprobe:do_sys_open dev:/sys/fs/bpf/kprobe-open  

2. Avoiding Hot Paths
Instrumentation on high-frequency paths (e.g., schedule(), do_IRQ()) can introduce significant overhead. Use tracepoint probes for these cases, as they are optimized for low-latency tracing.

3. Conditional Probing
Use kprobe_events to conditionally attach probes based on arguments or return values. For example:

# Trace sys_open only for files with specific numeric parameter  
sudo tee /etc/bpf/kprobe_events <<EOF  
kprobe:do_sys_open return <0x1000> {  
    if (PT_REGS_PARM1(ctx) == 0x1234)  
        bpf_printk("Found secret file!\n");  
}  
EOF

---

Symbol Resolution

1. Dynamic Symbol Resolution
BCC resolves symbols at runtime using kprobe_events or uprobe_events. For kernel symbols, ensure the kernel headers match the running kernel version. For user-space symbols, specify the address directly:

Example:

# Create uprobe_events file for user-space symbol resolution  
sudo tee /etc/bpf/uprobe_events <<EOF  
uprobe:/path/to/app:my_function /0x4005a0/  
EOF  

# Load and attach the program  
sudo bpftool prog load ./uprobe-resolve.o /sys/fs/bpf/uprobe-resolve  
sudo bpftool prog attach uprobe:/path/to/app:my_function dev:/sys/fs/bpf/uprobe-resolve  

2. Handling Symbol Clashes
If multiple probes target the same symbol, use a specific address to avoid conflicts. For example:

# Attach probe to a specific function address  
sudo tee /etc/bpf/uprobe_events <<EOF  
uprobe:/path/to/lib.so:my_function /0x4005a0/  
EOF

---

Avoiding Instrumentation Conflicts

1. Probe Overhead Mitigation
Use perf to check for existing probes and their impact:

# List existing kernel probes  
sudo perf list | grep 'kprobe'  

# Check probe count for a symbol  
sudo perf probe --list do_sys_open

2. Exclusive Probing
Avoid attaching multiple probes to the same function. Use --no-instrument in BCC to prevent accidental duplication:

# Prevent BCC from instrumenting a function  
bcc -l bpf -no-instrument -o myprog.o myprog.c

3. Debugging Conflicts
If probes fail to attach, check for symbol mismatches or permission issues:

# Verify symbol availability  
sudo nm /path/to/module.ko | grep 'my_function'  

---

Key takeaways

• Use kprobe for kernel functions and uprobe for user-space functions, prioritizing tracepoint for high-frequency paths.
• Resolve symbols dynamically by specifying addresses directly, and use perf to debug probe conflicts.
• Minimize overhead by avoiding redundant probes and leveraging BCC's --no-instrument flag to prevent accidental duplication.
• Always use sudo for bpftool commands and ensure elevated privileges for attaching probes to kernel or user-space targets.