apache / apache/arrow-rs

Push down (scalar) filters down to Parquet encoders

Open
#9,442 1 comment 2 reactions 0 assignees View on GitHub
performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently we always decode the

**Describe the solution you'd like**

We can push down certain operations (e.g. scalar equality / inequality) down to the (dictionary / RLE) encoders.
As we only have to consider the dictionary values, we can make this much more efficient than having to work on the decoded data (and e.g. avoiding creating string views).

In a quick test this seems saving 10-20% time on various queries in the `arrow_reader_clickbench` benchmarks.

**Describe alternatives you've considered**
**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the Parquet dictionary and RLE encoder paths, then inspect the arrow_reader_clickbench benchmarks mentioned in the issue. Determine where scalar equality and inequality filters currently cause decoded data to be created; done means supported filters are evaluated against encoder values and the benchmark behavior is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.