Compaction lag leads to blocks becoming unavailable
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
If compaction does not keep up with the ingestion rate, there is a chance that L3 compaction delay exceeds the threshold after which local blocks are removed from ingesters. Moreover, as long as queriers ignore blocks marked for removal, it's possible that a query plan will ignore all blocks for an "interval" completely (L1 as they are removed, L2 as they include duplicates, L3 as they are missing/incomplete). I think we should only mark blocks for removal if there is a child block available to queriers.
The main symptom is a gap in the data (all series) ending at `-querier.query-store-after` set for ingester (4h by default, 6h in the screenshot):
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.