Push down (scalar) filters down to Parquet encoders
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently we always decode the
**Describe the solution you'd like**
We can push down certain operations (e.g. scalar equality / inequality) down to the (dictionary / RLE) encoders.
As we only have to consider the dictionary values, we can make this much more efficient than having to work on the decoded data (and e.g. avoiding creating string views).
In a quick test this seems saving 10-20% time on various queries in the `arrow_reader_clickbench` benchmarks.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start with the Parquet dictionary and RLE encoder paths, then inspect the arrow_reader_clickbench benchmarks mentioned in the issue. Determine where scalar equality and inequality filters currently cause decoded data to be created; done means supported filters are evaluated against encoder values and the benchmark behavior is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100