apache / apache/datafusion

Build physical filter eagerly while constructing ParquetExec operator

Open
#2,059 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

We should avoid leaking logical expression to physical operators.

https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/physical_optimizer/pruning.rs#L93-L101

https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/physical_plan/file_format/parquet.rs#L77-L86

Contributor guide

Open the contributing guide

Research direction

Read datafusion/src/physical_optimizer/pruning.rs around lines 93-101 and datafusion/src/physical_plan/file_format/parquet.rs around lines 77-86. Trace how ParquetExec is constructed and verify that physical filter construction happens during operator construction without passing the logical expression into physical operators; run the relevant existing test suite to confirm behavior is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.