grafana / grafana/pyroscope

Compaction lag leads to blocks becoming unavailable

Open
#3,166 1 comment 1 reaction 1 assignee Claimed by @kolesnikovae View on GitHub
backend type/bug v1
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

If compaction does not keep up with the ingestion rate, there is a chance that L3 compaction delay exceeds the threshold after which local blocks are removed from ingesters. Moreover, as long as queriers ignore blocks marked for removal, it's possible that a query plan will ignore all blocks for an "interval" completely (L1 as they are removed, L2 as they include duplicates, L3 as they are missing/incomplete). I think we should only mark blocks for removal if there is a child block available to queriers.

The main symptom is a gap in the data (all series) ending at `-querier.query-store-after` set for ingester (4h by default, 6h in the screenshot):
image

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.