influxdata / influxdata/influxdb
feature proposal: online series file compaction
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
For ephemeral time series data, such as prometheus storage, the series cardinality is high but lots of series got deleted over time. Currently a series partition manages multiple segment files and one index file but the segments files are append-only. The deleted series are also appended as tombstone and the index file manages a tombstone map.
This result in both segment files and index file growing in size, and probably causes two consequences:
* more pagefaults when accessing the series entry
* cpu/memory overhead when rebuilding the index
PR https://github.com/influxdata/influxdb/pull/17185 added the functionality to compact series file offline but it requires influxdb server to be stopped, which is unfriendly in production.
So the proposal is to clean up the tombstones(deleted series) permanently during series partition compaction.
Contributor guide
Research direction
Start with PR #17185 and its offline series-file compaction implementation, then trace how series partitions handle segment files, the index file, and tombstone maps during compaction. Done means tombstones are cleaned up during online series partition compaction without requiring the InfluxDB server to stop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100