Wide schema performance: eliminate quadratic column-count scaling
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
Several independent code paths in the Parquet reader scale poorly with column count, leading to catastrophic performance on wide schemas. This epic-like issue tracks the general problem; individual PRs will reference it for context.
**To Reproduce**
N/A
**Expected behavior**
Parquet reader/writer operations and Arrow structures (RecordBatch, etc.) should scale linearly (ideally sub-linearly) with column count wherever practical.
**Additional context**
Contributor guide
Research direction
Start by mapping the independent Parquet reader and writer paths and the Arrow structures named in the issue, including RecordBatch. The issue provides no file names, tests, or reproduction; done means identifying and addressing the relevant quadratic column-count behavior so operations scale linearly or better where practical.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100