influxdata / influxdata/influxdb

Refactor `influxd inspect export-lp` to have more predictable resource usage

Open
#20,600 0 comments 0 reactions 0 assignees View on GitHub
area/2.x area/cli area/storage kind/tech-debt
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

The 2.x implementation of `export-lp` introduced in #20467 uses the `ReadAll` method on `TSMReader` to get the field-values for each exported series key. This matches the 1.x implementation, but was flagged during review as a lurking OOM for series keys with many fields.

Instead of `ReadAll`, we could use the `BlockIterator` method on `TSMReader` to process one block of data at a time. The new processing logic might need to copy some code out of `ReadAll` to ensure we're handling tombstones etc. in the same way.

Contributor guide

Open the contributing guide

Research direction

Start at the `influxd inspect export-lp` implementation and trace its use of `TSMReader.ReadAll`. Compare the `BlockIterator` path with `ReadAll`, preserving the existing tombstone handling while processing one block at a time. Done means export behavior remains equivalent without accumulating all field-values for a series key in memory.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.