Compactor: Improve storage of store-gateway meta blocks.
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Is your feature request related to a problem? Please describe.
Right now when the store-gateway starts, it download the bucket index, which contains the list of blocks for the current tenant.
The bucketindex is periodically created by the compactor.
The store gateway will download the each block meta using the bucket index at startup. However this leads to many object storage GETS to fetch all those meta every time we do a rollout.
#### Describe the solution you'd like
We could consider to use disk in helm to keep the storage-gateway [sync](https://github.com/grafana/pyroscope/blob/main/pkg/storegateway/bucket_stores.go#L44) folder or investigate how to store all meta in a new format with the bucket index. We need to check the size it would be for large cluster list ops and the time it would take to update this new files containing all meta.json.
I suggest we try to proto encode all blocks metas from ops and see how long it takes to encoded/decoded too.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.