influxdata / influxdata/influxdb

feature proposal: online series file compaction

Open
#20,968 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

For ephemeral time series data, such as prometheus storage, the series cardinality is high but lots of series got deleted over time. Currently a series partition manages multiple segment files and one index file but the segments files are append-only. The deleted series are also appended as tombstone and the index file manages a tombstone map.
This result in both segment files and index file growing in size, and probably causes two consequences:
* more pagefaults when accessing the series entry
* cpu/memory overhead when rebuilding the index

PR https://github.com/influxdata/influxdb/pull/17185 added the functionality to compact series file offline but it requires influxdb server to be stopped, which is unfriendly in production.

So the proposal is to clean up the tombstones(deleted series) permanently during series partition compaction.

Contributor guide

Open the contributing guide

Research direction

Start with PR #17185 and its offline series-file compaction implementation, then trace how series partitions handle segment files, the index file, and tombstone maps during compaction. Done means tombstones are cleaned up during online series partition compaction without requiring the InfluxDB server to stop.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.