grafana / grafana/pyroscope

Store-gateway memory usage

Open
#3,118 1 comment 2 reactions 1 assignee Claimed by @kolesnikovae View on GitHub
backend v1
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

In a large scale deployment, store-gateway service is prone to OOM issues due to the way we handle TSDB indices: once block is opened, its TSDB index is loaded in memory and never released until shutdown.

image

The issue is aggravated by the fact that the index:
- is not sharded: all series of a tenant block are stored in a single blob
- is not optimized for reading from the object storage: the whole index is to be loaded into memory before access

This makes it hard to cache the index efficiently: even if we pulled it from some very fast cache, we'd still need to have enough memory to keep indexes affected by the query in memory

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.