Investigate deduplication stream interleaving
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
When multiple replicas are queried, we deduplicate profiles in the k-way merge fashion. This results in that each of the replicas gets a list (chunked) of profiles to be read and processed.
The hypothesis is that the lists are not well aligned with the data placement, thus, we may need to scan larger data volumes.
For example, if we have 3 consistent replicas of 3 profile series s1, s2, s3 with 30 profiles each, we may access them in the following orger:
```
R1: s1[0:10), s2[0:10), s3[0:10),
R2: s1[10:20), s2[10:20), s3[10:20),
R3: s1[20:30), s2[20:30), s3[20:30),
```
In the meantime, we could plan the access in accordance to the data placement and could have less work to do:
```
R1: s1[0:30)
R2: s2[0:30)
R3: s3[0:30)
```
We still read 30 profiles from each of the replicas, but in the later case, we read them sequentially.
---
See https://github.com/grafana/pyroscope/blob/main/pkg/querier/select_merge.go#L242
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.