grafana / grafana/pyroscope

Compactor: Improve storage of store-gateway meta blocks.

Open
#2,592 1 comment 2 reactions 0 assignees View on GitHub
ops performance v1
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Is your feature request related to a problem? Please describe.

Right now when the store-gateway starts, it download the bucket index, which contains the list of blocks for the current tenant.
The bucketindex is periodically created by the compactor.

The store gateway will download the each block meta using the bucket index at startup. However this leads to many object storage GETS to fetch all those meta every time we do a rollout.

#### Describe the solution you'd like

We could consider to use disk in helm to keep the storage-gateway [sync](https://github.com/grafana/pyroscope/blob/main/pkg/storegateway/bucket_stores.go#L44) folder or investigate how to store all meta in a new format with the bucket index. We need to check the size it would be for large cluster list ops and the time it would take to update this new files containing all meta.json.

I suggest we try to proto encode all blocks metas from ops and see how long it takes to encoded/decoded too.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.