Build physical filter eagerly while constructing ParquetExec operator
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
We should avoid leaking logical expression to physical operators.
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/physical_optimizer/pruning.rs#L93-L101
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/physical_plan/file_format/parquet.rs#L77-L86
Contributor guide
Research direction
Read datafusion/src/physical_optimizer/pruning.rs around lines 93-101 and datafusion/src/physical_plan/file_format/parquet.rs around lines 77-86. Trace how ParquetExec is constructed and verify that physical filter construction happens during operator construction without passing the logical expression into physical operators; run the relevant existing test suite to confirm behavior is unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100