apache / apache/arrow-rs

Wide schema performance: eliminate quadratic column-count scaling

Open
#9,722 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**

Several independent code paths in the Parquet reader scale poorly with column count, leading to catastrophic performance on wide schemas. This epic-like issue tracks the general problem; individual PRs will reference it for context.

**To Reproduce**
N/A

**Expected behavior**

Parquet reader/writer operations and Arrow structures (RecordBatch, etc.) should scale linearly (ideally sub-linearly) with column count wherever practical.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by mapping the independent Parquet reader and writer paths and the Arrow structures named in the issue, including RecordBatch. The issue provides no file names, tests, or reproduction; done means identifying and addressing the relevant quadratic column-count behavior so operations scale linearly or better where practical.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.