grafana / grafana/pyroscope

Reduce unique stacktraces when merging profiles.

Open
#2,107 5 comments 1 reaction 1 assignee Claimed by @cyriltovena View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Is your feature request related to a problem? Please describe.

Currently when merging 3 hours of 200 pods within 16 applications, we can easily end up with a PProf files of 20MB. Interestingly, the size doesn't come from the symbols themselves. Symbols account for 10% of the size, the amount of unique stacktraces 1.2M is actually 90%.

This causes two issues:
- A query request duration can go up to 10s
- High CPU on queriers and ingester for allocating and merging those profiles.

#### Describe the solution you'd like

Since this is usually happening when merging profiles on a large cluster and for a long time range, we should definitely limits the amount of unique stacktraces that we are sending back.

I suggest we start first with limiting the amount of unique root function node we can returns. Unfortunately this will decrease the resolution of the merge, but large queries usually means you're looking at the big picture and that shouldn't disturb the investigation.

This will be added to the MergePprof API only for now with a default of 1k. (`MaxRootFunctions int64`).

In my experiment this has reduce the size by 80% while keeping the same flame graph in pprof:

```
=== RUN TestMerge
/Users/cyril/work/phlare/pkg/querier/querier_test.go:1067: Size:22 MB
len(fn): 30829
len(mapping): 16
len(locs): 175063
len(sample): 1156497
/Users/cyril/work/phlare/pkg/querier/querier_test.go:1102: Size:4.6 MB
len(fn): 13621
len(mapping): 7
len(locs): 30120
len(sample): 276819
--- PASS: TestMerge (4.20s)
```

In the future we can alwys trim nodes like pprof cli is doing with:

```
--nodecount=
Show at most so many nodes [default=80]
--nodefraction=
Hide nodes below *total [default=.005]
--edgefraction=
Hide edges below *total [default=.001]
```

The implementation can be found here https://github.com/google/pprof/blob/1710fef4ab10e4adc3c93771e7e0d9a1e174118b/internal/report/report.go#L124

This is a bit more tricky to implement, since you need to build the call tree first. But it has the advantage of keeping more granularity in the final result. For example if a stracktraces has large depth you can cut it down without altering the final total.

#### Describe alternatives you've considered

I thought about adding `other` as the example below:

![image](https://user-images.githubusercontent.com/1053421/223990573-e1a5f56d-7030-44e7-8a2e-d66eba1f599e.png)

I found it more disturbing than useful.

Since this is configurable limit at query time we can always increase it, and ultimately this can be used to avoid bad user behaviour in a cluster.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.