Reduce unique stacktraces when merging profiles.
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Is your feature request related to a problem? Please describe.
Currently when merging 3 hours of 200 pods within 16 applications, we can easily end up with a PProf files of 20MB. Interestingly, the size doesn't come from the symbols themselves. Symbols account for 10% of the size, the amount of unique stacktraces 1.2M is actually 90%.
This causes two issues:
- A query request duration can go up to 10s
- High CPU on queriers and ingester for allocating and merging those profiles.
#### Describe the solution you'd like
Since this is usually happening when merging profiles on a large cluster and for a long time range, we should definitely limits the amount of unique stacktraces that we are sending back.
I suggest we start first with limiting the amount of unique root function node we can returns. Unfortunately this will decrease the resolution of the merge, but large queries usually means you're looking at the big picture and that shouldn't disturb the investigation.
This will be added to the MergePprof API only for now with a default of 1k. (`MaxRootFunctions int64`).
In my experiment this has reduce the size by 80% while keeping the same flame graph in pprof:
```
=== RUN TestMerge
/Users/cyril/work/phlare/pkg/querier/querier_test.go:1067: Size:22 MB
len(fn): 30829
len(mapping): 16
len(locs): 175063
len(sample): 1156497
/Users/cyril/work/phlare/pkg/querier/querier_test.go:1102: Size:4.6 MB
len(fn): 13621
len(mapping): 7
len(locs): 30120
len(sample): 276819
--- PASS: TestMerge (4.20s)
```
In the future we can alwys trim nodes like pprof cli is doing with:
```
--nodecount=
Show at most so many nodes [default=80]
--nodefraction=
Hide nodes below *total [default=.005]
--edgefraction=
Hide edges below *total [default=.001]
```
The implementation can be found here https://github.com/google/pprof/blob/1710fef4ab10e4adc3c93771e7e0d9a1e174118b/internal/report/report.go#L124
This is a bit more tricky to implement, since you need to build the call tree first. But it has the advantage of keeping more granularity in the final result. For example if a stracktraces has large depth you can cut it down without altering the final total.
#### Describe alternatives you've considered
I thought about adding `other` as the example below:

I found it more disturbing than useful.
Since this is configurable limit at query time we can always increase it, and ultimately this can be used to avoid bad user behaviour in a cluster.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.