Add selectivity threshold for filter pushdown
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The parquet reader doesn't have a way to skip pushing down filters that don't filter out a lot of values.
When most values match the filter (e.g. 30-50%) we should just materialize everything instead and avoid the overhead of the interleaved read/skip calls.
**Describe the solution you'd like**
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
The issue names only the parquet reader; start by locating its filter-pushdown entry point and any existing selectivity or reader tests. The requested threshold and solution are not specified, so first determine how selectivity should be measured and what tests would define avoiding interleaved read/skip calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100