grafana / grafana/pyroscope

Investigate deduplication stream interleaving

Open
#2,546 1 comment 1 reaction 0 assignees View on GitHub
backend performance storage v1
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

When multiple replicas are queried, we deduplicate profiles in the k-way merge fashion. This results in that each of the replicas gets a list (chunked) of profiles to be read and processed.

The hypothesis is that the lists are not well aligned with the data placement, thus, we may need to scan larger data volumes.

For example, if we have 3 consistent replicas of 3 profile series s1, s2, s3 with 30 profiles each, we may access them in the following orger:

```
R1: s1[0:10), s2[0:10), s3[0:10),
R2: s1[10:20), s2[10:20), s3[10:20),
R3: s1[20:30), s2[20:30), s3[20:30),
```

In the meantime, we could plan the access in accordance to the data placement and could have less work to do:
```
R1: s1[0:30)
R2: s2[0:30)
R3: s3[0:30)
```

We still read 30 profiles from each of the replicas, but in the later case, we read them sequentially.

---

See https://github.com/grafana/pyroscope/blob/main/pkg/querier/select_merge.go#L242

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.