iovisor / iovisor/bcc

kretprobes are mysteriously missed

Open
#2,825 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
10d 4h
Merged PRs (30d)
3

Description

I have a BCC / python script that aims to measure the time that is taken by FUSE I/O requests. For this purpose it sets a kprobe and a kretprobe on (among others, but this the one I have trouble with), `fuse_file_write_iter`. I set the kretprobe before the kprobe:

```
maxactive = multiprocessing.cpu_count() * 5
b.attach_kretprobe(event="fuse_file_write_iter", fn_name="trace_write_return", maxactive=maxactive)
b.attach_kprobe(event="fuse_file_write_iter", fn_name="trace_write_entry")
```
However is seems that the kretprobe is called significantly less often than the kprobe, which causes my code (which counts enters and exits) to think that some I/O requests are ongoing forever.

I see this from these counts in `/sys/kernel/debug/tracing/kprobe_profile`:
```
/sys/kernel/debug/tracing# cat kprobe_profile|grep 33922|sort -k 2 -n
r_fuse_file_write_iter_bcc_33922 74381756 0
p_fuse_file_write_iter_bcc_33922 74382094 0
```
As you can see, the number of caught returns is 338 lower than the number of enters, but `nmissed` is `0`. So the fact that events are lost is lost too. Something is very unreliable here. What?

As a second question: I understand from reading about the implementation of kretprobes that a kretprobe is actually a special kprobe which causes the return from the function to trigger some code as well. You can even have code of your own at the entry, as I understand from `https://www.kernel.org/doc/Documentation/kprobes.txt` (section retprobe entry-handler). But it seems that this facility is not available from python-bcc. It would be exellently suited to ensure that entries and exits from a function balance properly. By splitting that up in 2 probes, the guarantee is lost, and it is no doubt less efficient.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the two attachments for fuse_file_write_iter and compare their counts in /sys/kernel/debug/tracing/kprobe_profile. Read the referenced kprobes documentation and the Python BCC attachment paths to determine why returns differ and whether retprobe entry handlers are exposed; done means a confirmed root cause and a clearly scoped fix or documentation change.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
observability-sre, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.