Parquet reader should return a RowSelection to support result caching
Open
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
When the same predicate is applied to the same Parquet file, the resulting row selection is deterministic. Returning and caching the RowSelection allows downstream consumers to reuse it, avoiding redundant filtering and improving read performance.
I believe this feature will require changes in both arrow-rs and datafusion.
Contributor guide
Research direction
Start by tracing the Parquet reader and RowSelection handling in arrow-rs and DataFusion. Determine how the same predicate and file can expose a reusable selection to downstream consumers, then verify that the selection can be returned and cached without changing query results or filtering behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100