grafana / grafana/pyroscope

ebpf: Dropping samples without a stack distorts flamegraphs

Open
#2,026 2 comments 2 reactions 0 assignees View on GitHub
eBPF
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

In noticed that samples without a stack are simply dropped instead of being processed (thereby accounting the CPU time spent to the process):

https://github.com/grafana/phlare/blob/11fcb2415e50839b06a827eee70b5a3ccc6f1834/ebpf/session.go#L173

In my opinion, it would be better to still process these comm-only samples as you otherwise get a quite distorted picture when looking at profiles.

For example, kernel threads frequently perform work asynchronously on behalf of user processes. If those samples are simply dropped (if someone only collects user stacks), a user might end up wondering why the CPU time reported from other tools is much larger than the CPU time reported in Pyroscope. Even if this work can not be easily traced back to the application (unless you just run one application, which is common), it is still useful for context and to prevent confusion.

Specifically, you can not currently
- see how much time a process spent inside/outside of kernel/user space if the collection of the respectively other stacks is disabled (e.g., only collecting kernel stacks, you can not see if the process actually spends a significant amount of it's time in the kernel).
- see which other user/kernel processes consumed CPU time (e.g, only collecting user stacks, you do not see if there are any kernel threads that actually do all the work, which is common in disk IO, while your user process only does little).

Also, can we assume that the kernel helper functions always give you at least one stack frame if the process is currently in user/kernel space? If this is not the case, the picture is further distorted. Even if you not know the function, it is still useful to know that the CPU time was spent in that process.

To be sure: If samples are dropped here, does this still allow for an accurate calculation of the CPU time information in the flamegraphs? I.e., is the CPU time per sample calculated as count*(total_time/total_count) or is the amount of CPU time for a sample calculated as count*(1s/sampling_rate)? Because of https://github.com/grafana/agent/blob/e3cbf93f47311a4044b9a190adae23f74eb6d654/component/pyroscope/ebpf/ebpf_linux.go#L160 I assume it's the latter.

Other relevant code locations:
- Processing of the sample in the agent's callback function:
https://github.com/grafana/agent/blob/e3cbf93f47311a4044b9a190adae23f74eb6d654/component/pyroscope/ebpf/ebpf_linux.go#L164
- Invocation of the callback function:
https://github.com/grafana/phlare/blob/11fcb2415e50839b06a827eee70b5a3ccc6f1834/ebpf/session.go#L176

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.