kretprobes are mysteriously missed
- Dominant language
- C
- Stars
- 22.7k
- Forks
- 4.1k
- Avg merge
- 10d 4h
- Merged PRs (30d)
- 3
Description
I have a BCC / python script that aims to measure the time that is taken by FUSE I/O requests. For this purpose it sets a kprobe and a kretprobe on (among others, but this the one I have trouble with), `fuse_file_write_iter`. I set the kretprobe before the kprobe:
```
maxactive = multiprocessing.cpu_count() * 5
b.attach_kretprobe(event="fuse_file_write_iter", fn_name="trace_write_return", maxactive=maxactive)
b.attach_kprobe(event="fuse_file_write_iter", fn_name="trace_write_entry")
```
However is seems that the kretprobe is called significantly less often than the kprobe, which causes my code (which counts enters and exits) to think that some I/O requests are ongoing forever.
I see this from these counts in `/sys/kernel/debug/tracing/kprobe_profile`:
```
/sys/kernel/debug/tracing# cat kprobe_profile|grep 33922|sort -k 2 -n
r_fuse_file_write_iter_bcc_33922 74381756 0
p_fuse_file_write_iter_bcc_33922 74382094 0
```
As you can see, the number of caught returns is 338 lower than the number of enters, but `nmissed` is `0`. So the fact that events are lost is lost too. Something is very unreliable here. What?
As a second question: I understand from reading about the implementation of kretprobes that a kretprobe is actually a special kprobe which causes the return from the function to trigger some code as well. You can even have code of your own at the entry, as I understand from `https://www.kernel.org/doc/Documentation/kprobes.txt` (section retprobe entry-handler). But it seems that this facility is not available from python-bcc. It would be exellently suited to ensure that entries and exits from a function balance properly. By splitting that up in 2 probes, the guarantee is lost, and it is no doubt less efficient.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the two attachments for fuse_file_write_iter and compare their counts in /sys/kernel/debug/tracing/kprobe_profile. Read the referenced kprobes documentation and the Python BCC attachment paths to determine why returns differ and whether retprobe entry handlers are exposed; done means a confirmed root cause and a clearly scoped fix or documentation change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python
- Domain
- observability-sre, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100