Enable parquet filter pushdown (`filter_pushdown`) by default
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
In https://github.com/apache/arrow-datafusion/pull/3380 @thinkharderdev added support for evaluating filters during the parquet scan via the RowIndex mechanism 🎉
This feature is currently enabled via a feature flag, which is disabled by default.
This ticket tracks enabling this feature by default.
Currently known items are:
- [x] https://github.com/apache/arrow-datafusion/issues/3457
- [x] Review of parquet test coverage and add additional coverage as needed: https://github.com/apache/arrow-datafusion/pull/3976
- [x] https://github.com/apache/arrow-datafusion/issues/4005
- [x] https://github.com/apache/arrow-datafusion/issues/4006
- [x] https://github.com/apache/arrow-datafusion/issues/4046
- [ ] Enable this feature by default: https://github.com/apache/arrow-datafusion/pull/3828
Contributor guide
Research direction
Start by reviewing PR 3828 and the completed prerequisite issues listed in the ticket, especially the parquet test-coverage work in PR 3976. Enable filter_pushdown by default, then verify the existing parquet test coverage and confirm the feature no longer requires an opt-in flag.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100