apache / apache/datafusion

Avoid evaluating filters when they can be discarded purely from statistics

Open
#15,425 4 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently stats filter pruning (both at the row group and page level) has one of two outcomes per container:
1. This container cannot possibly match the filter (discard it).
2. This container *may* match the filter, but which rows to include or exclude needs to be confirmed by evaluating each row of the data.

There is a big optimization here which is *if we know that every row in the container matches the filter, we don't need to evaluate the filter at all*.

Consider a column `name` with values `["Adrian", "Adrian", "Adrian"]`. The min/max stats are `"Adrian"/"Adrian"`. A query with the filter `name = "Adrian"` should not need to ever read the column to know that all rows match the filter.

Another relevant case is a `ts` column with values `["2025-01-01T00:00:00Z", ..., "2025-01-01T00:01:32Z"]`. The values need not be sorted or ordered, but let's say that the min/max stats are `"2025-01-01T00:00:00Z"/"2025-01-01T00:01:32Z"`. For a filter `ts > '2024-12-31T00:00:00Z'` there should be no need to evaluate the filter on every row: we know just from stats that every row matches.

We could incorporate this change, but it would require some refactoring of https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pruning.rs and consumers.

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/physical-optimizer/src/pruning.rs and trace the row-group and page-level pruning consumers. Determine how statistics currently distinguish discarded containers from containers requiring row evaluation; done means containers proven to fully match no longer evaluate their filters, while existing pruning behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.