Enabling a probe that's in many locations overflows the instruction limit
- Dominant language
- C
- Stars
- 22.7k
- Forks
- 4.1k
- Avg merge
- 10d 4h
- Merged PRs (30d)
- 3
Description
This might be tangentially related to issue #1774
I have a shared library with a dozen or so labels for static probes (via DTRACE_PROBE2()), with 1 to ~3500 probes of a given label currently. If I try to do `u.enable_probe(probe="OTHER", fn_name="do_fn")` it fails if the probe name is for the 3500 instance case, and works for the (much smaller) other ones.
This is the error: `bpf: Invalid argument. Program too large (43943 insns), at most 4096 insns`
I'm presuming reading between the lines of issue #1774 that instead of creating one program for each probe it's attached to, it creates a single program that has effectively a huge case statement at the start.
Options would seem to be:
1) one program per probe - con, expensive in memory (if that matters here), expensive in compiling/jit-ing N programs instead of one more-complex program - not sure how bad that really is, but it's more CPU for sure. pros: simple (and perhaps could be an easy option to add invoked by the user, as a stopgap)
2) detect program-space overflows caused by this expansion and duplicate only at that point. Cons: more complex during compilation (need to approximate available space before inserting the case statement and check after each insertion, or something like that (depends on internal structure). Pros: minimal added memory or CPU, and then only in cases which *need* it.
3) Put an arbitrary limit before overflowing to another copy, and allow the user to tweak the limit if it still overflows (i.e. big program, or case statement more expensive than normal for some reason). Cons: may use a bit more memory/cpu than #2. Pros: simple, rarely would use any more memory.
Any of these would help avoid people contorting their probes to avoid "too many" with the same name and manually duplicating their program (`u.enable_probe(probe="group1"...); u.enable_probe(probe="group2",...) etc` It would also help avoid "my bcc program worked and then someone added a couple more probe points and now it fails".
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the u.enable_probe(probe=..., fn_name=...) path and the behavior discussed in issue #1774, then trace how one program expands across many matching probes. Done means enabling a label with roughly 3,500 instances no longer fails with the 4,096-instruction BPF limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100