grafana / grafana/pyroscope

Reclaim metastore BoltDB disk space without requiring restart

Open
#5,279 0 comments 0 reactions 0 assignees View on GitHub
backend enhancement ops performance storage
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

## Problem

Metastore index cleanup can delete old index buckets and add tombstones, but the underlying BoltDB file does not shrink in a long-running process. Disk space is only reclaimed when a snapshot is restored and `metastore.snapshot-compact-on-restore=true`.

In practice, this means long-running single-binary/self-hosted deployments can keep a large `metastore.boltdb` file even after retention cleanup has removed old data from the logical index.

This is especially problematic for local filesystem deployments with finite disk, where the process may run until disk exhaustion and crashloop before compaction ever happens.

## Code pointers

- `pkg/metastore/fsm/boltdb.go`
- `bbolt.Compact` is only called from the snapshot restore path.
- compaction is gated by `SnapshotCompactOnRestore`.
- `pkg/metastore/fsm/fsm.go`
- `metastore.snapshot-compact-on-restore` defaults to `false`.
- snapshots are persisted from the running BoltDB state, but regular online compaction is not performed.
- `pkg/metastore/index_raft_handler.go`
- `TruncateIndex` deletes logical index shards, but this does not shrink the BoltDB file.

## Proposal

Add a mechanism to reclaim metastore BoltDB disk space for long-running metastores without requiring a restart/snapshot restore cycle.

Possible approaches:

- Periodically compact the BoltDB database in the background when safe.
- Compact after successful snapshots outside of the restore path.
- Provide an explicit admin/API operation to trigger metastore DB compaction.
- Track BoltDB file size vs logical/live size and compact only above a threshold.

## Acceptance criteria

- A long-running metastore can reclaim BoltDB disk space after retention cleanup without process restart.
- The mechanism does not block raft apply/read paths for excessive time.
- The mechanism is safe for single-node and multi-node raft deployments.
- Metrics/logging expose when compaction runs, how long it takes, and compaction ratio.
- Add tests for compaction behavior or at least the triggering/gating logic.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading pkg/metastore/fsm/boltdb.go, pkg/metastore/fsm/fsm.go, and pkg/metastore/index_raft_handler.go, then inspect existing metastore tests. Trace snapshot restore compaction and TruncateIndex behavior before choosing a safe trigger. Done means online space reclamation works for single- and multi-node Raft deployments without excessive blocking, with compaction metrics/logging and tests for the trigger or gating logic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.