Implement expression simplification for filtering (vs generically)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
When simplifying expressions in general, care must be taken with nulls. For example it seems like this expression can be simplified to `false`:
```sql
A AND !A
```
however, if `A` is `null` then the expression actually evaluates to `null` and not `false`
The existing simplification code handles these cases by checking `is_nullable`:
https://github.com/apache/arrow-datafusion/blob/71efcf5ee8900a1efe12fb812e210e6941060733/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs#L575-L582
However, if we are using `A AND !A` as a filter, we can actually simplify the whole thing to `false` because for filtering:
* `true` --> row is kept
* `false` --> row is not kept
* `null` --> row is not kept (same as false)
### Describe the solution you'd like
I would like someone to figure out how to take advantage of the above observation to apply more simplification rules when simplifying predicates
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs around the existing is_nullable handling. Compare general expression simplification with the three-valued filtering semantics described in the issue, then determine which predicate-specific rules preserve filter behavior. Done means the design and implementation simplify eligible filter predicates without changing retained rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100