6 minutes
How Do I Profile eBPF Code?
If we are running any eBPF workload or writing eBPF code, we want to measure its performance impact, and in this post we will demonstrate an example of how to do it.
In this example, our goal is to measure the performance of file open operations, one of the most critical functions in the OS. Our code used file open hooks in eBPF, we wanted to measure the performance overhead introduced by adding this hook.
To identify likely bottlenecks, we need a simple C test harness with the fewest dependencies, designed to measure file open performance.
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
#include
#include
static
inline
uint64_t
now_ns
(
void
) {
struct
timespec ts;
clock_gettime
(CLOCK_MONOTONIC,
&
ts);
/* VDSO, no syscall */
return
(
uint64_t
)ts.tv_sec
*
1000000000ull
+
ts.tv_nsec;
}
int
main
(
int
argc,
char
**
argv) {
const
char
*
path
=
argv[
1
];
uint64_t
n
=
strtoull
(argv[
2
], NULL,
10
);
uint64_t
warm
=
n
/
10
;
/* preallocate + prefault + lock: no page faults in the loop */
uint32_t
*
d
=
mmap
(NULL, n
*
sizeof
(
uint32_t
), PROT_READ
|
PROT_WRITE,
MAP_PRIVATE
|
MAP_ANONYMOUS
|
MAP_POPULATE,
–
1
,
0
);
mlock
(d, n
*
sizeof
(
uint32_t
));
for
(
uint64_t
i
=
0
; i
<
n; i
++
) d[i]
=
0
;
/* fault everything in */
for
(
uint64_t
i
=
0
; i
=
0
)
close
(fd);
d[i]
=
(
uint32_t
)(t1
–
t0);
}
/* dump after the loop only */
for
(
uint64_t
i
=
warm; i
/tmp/samples.txt
The
taskset -c 3
pins execution to
CPU 3
, reducing CPU migration noise, and
chrt -f 99
gives the benchmark extremely high CPU priority. It runs before almost all normal programs and keeps running until it finishes, blocks, or is interrupted. The C code discards the first 10%; the file should contain 90,000 samples.txt.
Next, run the eBPF code and execute something like this.
sudo $PERF record
\
-g
\
–call-graph fp
\
-e cycles:k
\
-F
997
\
-o ~/perf.data
\
—
\
taskset -c
3
\
chrt -f
99
\
./bench /etc/hostname
200000
\
> /tmp/samples.txt
The
-g
records the call stacks,
–call-graph fp
unwinds stacks using frame pointers, and
-e
samples CPU cycles in kernel mode only, which includes syscall, VFS, LSM, and eBPF execution and not userspace benchmark work. The
-F 997
requests 997 samples per second, and the non-round frequency helps avoid periodic alignment.
After running the above, run this command to sort the data.
sudo
”
$PERF
”
report -i ~/perf.data –stdio –sort comm,dso,symbol > perf.txt
Flamegraph of the same perf.data, generated with Inferno. The stack of interest here is bpf_lsm_file_open and everything above it.
Here is an example output from perf.txt, which shows that the time being spent on bpf_lsm_file_open and its tail calls is where the performance bottleneck is. This turned out to be in a hot path, which meant every allocation-to-CPU cycle shaving will make a significant impact on the performance of the system.
|
| | | | | |–90.52%–do_dentry_open
| | | | | | |
| | | | | | –89.78%–bpf_lsm_file_open
| | | | | | |
| | | | | | –89.30%–0xffffffffc0288c18
| | | | | | |
| | | | | | |–87.40%–bpf_prog_b06f413955402a4b_tail_call_security_check
| | | | | | | |
| | | | | | | |–77.57%–bpf_prog_934361d723613c1c_enforce_access_policy
| | | | | | | | |
| | | | | | | | |–57.87%–bpf_prog_a0f18f4b0b140d77_path_check_callback
| | | | | | | | | |
| | | | | | | | | |–29.94%–bpf_probe_read_kernel
| | | | | | | | | | |
| | | | | | | | | | |–18.88%–copy_from_kernel_nofault
| | | | | | | | | | |
| | | | | | | | | | –4.99%–copy_from_kernel_nofault_allowed
| | | | | | | | | |
| | | | | | | | | |–4.88%–htab_map_hash
| | | | | | | | | |
| | | | | | | | | –2.29%–copy_from_kernel_nofault
This post focuses on the profiling method rather than a specific result, and the overhead you’ll see depends heavily on what your hook actually does, so we’re leaving the numbers out and focusing on how to get them yourself.
Now, from the above, we can start analyzing where the time is being spent and perf-tune the code along with the p50/p99 of the C code with eBPF running.
By doing this, we can clearly identify the perf impact of the eBPF code and likely pinpoint where optimizations are required. It can be as simple as caching something or coming up with a better algorithm based on where the problem is.
eBPF
1118 Words
2026-07-22 21:47 +0000
Read other posts
When LLVM’s Optimizer Breaks Your eBPF Program
→