apache / apache/arrow-rs

Column performance: run-proportional read/write cost

Open
#9,731 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**

Several Parquet read and write paths allocate memory and perform computation proportional to the total row count, even when most values are repetitions, or null. For a column that is 99% null, this means the current implementation sometime does ~100x more work than necessary. This issue tracks the general problem; individual PRs will reference it for context.

**To Reproduce**
N/A

**Expected behavior**
For Parquet file storing definition and repetition levels in RLE encoding, the cost should be proportional to the number of runs.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the Parquet read and write paths that process definition and repetition levels in RLE encoding. Measure where work and allocations scale with total rows rather than runs, including columns with many null or repeated values. Done means the affected paths have cost proportional to the number of runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.