Store-gateway memory usage
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
In a large scale deployment, store-gateway service is prone to OOM issues due to the way we handle TSDB indices: once block is opened, its TSDB index is loaded in memory and never released until shutdown.
The issue is aggravated by the fact that the index:
- is not sharded: all series of a tenant block are stored in a single blob
- is not optimized for reading from the object storage: the whole index is to be loaded into memory before access
This makes it hard to cache the index efficiently: even if we pulled it from some very fast cache, we'd still need to have enough memory to keep indexes affected by the query in memory
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.