apache / apache/arrow-rs

Add selectivity threshold for filter pushdown

Open
#9,415 0 comments 2 reactions 0 assignees View on GitHub
performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The parquet reader doesn't have a way to skip pushing down filters that don't filter out a lot of values.
When most values match the filter (e.g. 30-50%) we should just materialize everything instead and avoid the overhead of the interleaved read/skip calls.

**Describe the solution you'd like**

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

The issue names only the parquet reader; start by locating its filter-pushdown entry point and any existing selectivity or reader tests. The requested threshold and solution are not specified, so first determine how selectivity should be measured and what tests would define avoiding interleaved read/skip calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.