apache / apache/datafusion

EPIC: use cp_solver framework to develop a more sophisticated predicate simplification

Open
#16,511 3 comments 1 reaction 0 assignees View on GitHub
enhancement PROPOSAL EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The predicates in filter can be variable, it's possible to use the cp_solver to simplify the predicates in filter, then reduce the computing cost for filter.

### Describe the solution you'd like

We've done some basic simplification in https://github.com/apache/datafusion/pull/16362, we can make it a start to develop a more sophisticated rule to cover more cases

### Describe alternatives you've considered

_No response_

### Additional context

What I want to discuss:
1. Make it a new physical optimizer rule? Or add them to the current physical filter push down
2. Some filters, such as partition columns related predicates, will be fully pushed down scan, that is, we don't have such predicates in some FilterExec. How to capture this?
3. TBD (You may have other thoughts!)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the basic predicate simplification in pull request 16362 and the physical filter pushdown path, including FilterExec. The work first requires deciding whether this belongs in a new physical optimizer rule or existing filter pushdown, and how to handle predicates fully pushed into scans; done means an agreed approach that covers more simplification cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.