grafana / grafana/pyroscope

Inlined frames are skipped

Open
#4,704 3 comments 0 reactions 1 assignee Claimed by @korniltsev-grafanista View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Describe the bug

During symbolication inlined frames are skipped, resulting in confusing profiles.

There's lots of inlining in real world software (especially with LTO).

#### To Reproduce

Compile this:

```c
#include

int innie() {
for (;;) {
printf(".");
}

return 0;
}

int middle() {
return innie();
}

int __attribute__ ((noinline)) outie() {
return middle();
}

int main() {
return outie();
}
```

And run it on a host covered by Grafana Alloy with `pyroscope.ebpf` enabled.

Observe profiles missing `middle`:

Image

If you try running `addr2line` manually with inlines requested, you'd see this:

```
ivan@cube:~$ llvm-addr2line -ifa -e derp 0x1164
0x1164
middle
/home/ivan/./derp.c:12
outie
/home/ivan/./derp.c:16
```

#### Expected behavior

Inline frames are present in profiles.

#### Additional Context

You can see that multiple frames can be returned, but only one is used:

https://github.com/grafana/opentelemetry-ebpf-profiler/blob/f910fc4db23e6552ed79507de3788f660a9471bc/pyroscope/symb/irsymcache/table.go#L45-L49

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.