Column performance: run-proportional read/write cost
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
Several Parquet read and write paths allocate memory and perform computation proportional to the total row count, even when most values are repetitions, or null. For a column that is 99% null, this means the current implementation sometime does ~100x more work than necessary. This issue tracks the general problem; individual PRs will reference it for context.
**To Reproduce**
N/A
**Expected behavior**
For Parquet file storing definition and repetition levels in RLE encoding, the cost should be proportional to the number of runs.
**Additional context**
Contributor guide
Research direction
Start by locating the Parquet read and write paths that process definition and repetition levels in RLE encoding. Measure where work and allocations scale with total rows rather than runs, including columns with many null or repeated values. Done means the affected paths have cost proportional to the number of runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100