apache / apache/datafusion

Parquet reader should return a RowSelection to support result caching

Open
#24,062 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

When the same predicate is applied to the same Parquet file, the resulting row selection is deterministic. Returning and caching the RowSelection allows downstream consumers to reuse it, avoiding redundant filtering and improving read performance.

I believe this feature will require changes in both arrow-rs​ and datafusion.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Parquet reader and RowSelection handling in arrow-rs and DataFusion. Determine how the same predicate and file can expose a reusable selection to downstream consumers, then verify that the selection can be returned and cached without changing query results or filtering behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.