apache / apache/arrow-rs

Dictionary page pruning for row filter predicates

Open
#9,588 0 comments 0 reactions 0 assignees View on GitHub
parquet performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

## Summary

When a predicate targets a dictionary-encoded column, evaluate the predicate against the dictionary values before decoding any data pages. If no dictionary values match, the entire column chunk can be skipped (`AllFalse`). If all match, per-row evaluation can be skipped (`AllTrue`).

This adds an `evaluate_dictionary` method to the `ArrowPredicate` trait and integrates dictionary pruning into the row group filter evaluation pipeline.

Related PR: #9574

Contributor guide

Open the contributing guide

Research direction

Start with the ArrowPredicate trait and the row group filter evaluation pipeline described in the issue; inspect how dictionary values and data pages are currently handled. The work is done when dictionary predicates produce AllFalse or AllTrue where applicable and avoid unnecessary per-row evaluation, with behavior covered by the relevant filter tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.