EPIC: use cp_solver framework to develop a more sophisticated predicate simplification
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
The predicates in filter can be variable, it's possible to use the cp_solver to simplify the predicates in filter, then reduce the computing cost for filter.
### Describe the solution you'd like
We've done some basic simplification in https://github.com/apache/datafusion/pull/16362, we can make it a start to develop a more sophisticated rule to cover more cases
### Describe alternatives you've considered
_No response_
### Additional context
What I want to discuss:
1. Make it a new physical optimizer rule? Or add them to the current physical filter push down
2. Some filters, such as partition columns related predicates, will be fully pushed down scan, that is, we don't have such predicates in some FilterExec. How to capture this?
3. TBD (You may have other thoughts!)
Contributor guide
Research direction
Start by reviewing the basic predicate simplification in pull request 16362 and the physical filter pushdown path, including FilterExec. The work first requires deciding whether this belongs in a new physical optimizer rule or existing filter pushdown, and how to handle predicates fully pushed into scans; done means an agreed approach that covers more simplification cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100