influxdata / influxdata/influxdb
Refactor `influxd inspect export-lp` to have more predictable resource usage
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
The 2.x implementation of `export-lp` introduced in #20467 uses the `ReadAll` method on `TSMReader` to get the field-values for each exported series key. This matches the 1.x implementation, but was flagged during review as a lurking OOM for series keys with many fields.
Instead of `ReadAll`, we could use the `BlockIterator` method on `TSMReader` to process one block of data at a time. The new processing logic might need to copy some code out of `ReadAll` to ensure we're handling tombstones etc. in the same way.
Contributor guide
Research direction
Start at the `influxd inspect export-lp` implementation and trace its use of `TSMReader.ReadAll`. Compare the `BlockIterator` path with `ReadAll`, preserving the existing tombstone handling while processing one block at a time. Done means export behavior remains equivalent without accumulating all field-values for a series key in memory.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100