apache / apache/datafusion

Support predicate pruning for basic transformations such as `upper`

Open
#14,054 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Follow up to #507.

Predicate pruning is a powerful technique to speed up queries by skipping entire files / pieces of work based on summary statistics of the data.

This issue proposes implementing predicate pruning for expressions such as `lower(col) = 'abc'`. The idea is that if we have a min stat such as `AbC` we should be able to transform it to `'abc'` and push down the predicate (in this case it might match). Or given the min/max `YYY`/`ZZZ` then `lower(col) = 'abc'` could never match so the file can be skipped.

To implement this you'll need to make a PR similar to https://github.com/apache/datafusion/pull/12978 and add fuzz tests (see https://github.com/apache/datafusion/pull/13253).

One thing to think about is how we can make this work in concert with other predicate push down. That is, it would be ideal if something like this could be pushed down: `lower(col) like 'abc%'`. That may require a lot of refactoring and might need to be done in a series of PRs, an initial PR that just implements the `=` case would be a good start to prove that it's possible. But it may also be worth exploring generalizing e.g. `lower(col) like 'abc%'` becomes `col ilike 'abc%'` which we then push down? A discussion of pros and cons is warranted.

Contributor guide

Open the contributing guide

Research direction

Start by reading issue #507, the approach in PR #12978, and the fuzz-test patterns from PR #13253. Define an initial equality-only slice for transformations such as lower or upper, then verify that predicate pruning can use transformed statistics and add the requested fuzz coverage; broader LIKE and refactoring are explicitly follow-up considerations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.