grafana / grafana/pyroscope

metastore: query historical backfills outside the partition lookaround window

Open
#5,361 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

## Problem

The metastore index partitions shards by the block ULID timestamp, which represents when the block was created. Metadata queries previously selected partitions by the requested profile time, expanded by a fixed 24-hour lookaround period.

This fails for historical backfills: a block created today can contain profiles from weeks or months ago. Queries for the historical profile range do not inspect today's partition, so the backfilled data is not returned.

The same mismatch affects retention. A partition can be old while containing recently timestamped profile data.

## Expected behavior

Metadata queries must find blocks based on the profile data interval (`BlockMeta.MinTime` / `MaxTime`), regardless of the block creation timestamp or its partition.

Retention must preserve shards containing data newer than the retention marker, even when their partition timestamp is old.

## Proposed approach

- Build an in-memory interval index from persisted `ShardIndex` min/max summaries.
- Use it to select candidate tenant shards for metadata and label queries.
- Fall back to scanning Bolt for stale read snapshots and pending shard deletions.
- Keep compatibility with legacy shard summaries that do not have a time range.
- Evaluate retention using `ShardIndex.MaxTime` as milliseconds.

## Reproduction

1. Upload or ingest a profile with a timestamp more than 24 hours in the past.
2. Query a narrow interval around that historical timestamp.
3. The query returns no data despite the block existing in the metastore.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the metastore metadata and label query paths, retention handling, and persisted ShardIndex summaries described in the issue. Verify the historical-backfill reproduction, then confirm queries select shards by BlockMeta.MinTime/MaxTime and retention preserves shards using ShardIndex.MaxTime, including legacy summaries and stale snapshots.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.