apache / apache/datafusion

[EPIC] Simplify datetime predicates using "preimages"

Open
#19,946 1 comment 1 reaction 0 assignees View on GitHub
enhancement EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

It is much easier for optimizers to reason about predicates of the form ` op `expressions. They often can't optimize anywhere near as well if they have a scalar function wrapping them

This includes DataFusion's [PruningPredicate](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html#contains-analysis-and-minmax-rewrite)

For example the predicate looking for a particular year
```sql
WHERE EXTRACT (YEAR FROM k) = 2024
```

Can be rewritten as
```sql
k >= 2024-01-01 AND k < 2025-01-01.
```

And then k is easier to pushdown and subject to range analysis, etc.

The ClickHouse paper : https://www.vldb.org/pvldb/vol17/p3731-schulze.pdf calles these "preimage" (from the [mathematical term](https://en.wikipedia.org/wiki/Image_(mathematics))) for this rewrite (I think toYear(k) is the equivalent of EXTRACT(YEAR from k))

> Second, some functions can compute the preimage of a given function result. This is used to replace comparisons of constants with function calls on the key columns by comparing the key column value with the preimage. For example, toYear(k) = 2024 can be replaced by k >= 2024-01-01 && k < 2025-01-01.

This ticket tracks adding such optimziations to DataFusion

### Describe the solution you'd like

- [x] https://github.com/apache/datafusion/issues/18320
- [x] https://github.com/apache/datafusion/issues/19889
- [x] https://github.com/apache/datafusion/pull/20059
- [ ] https://github.com/apache/datafusion/issues/18319
- [x] https://github.com/apache/datafusion/pull/20050
- [ ] https://github.com/apache/datafusion/issues/20197

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing DataFusion's PruningPredicate documentation and the remaining checklist issues 18319 and 20197. Compare those open items with the completed issues and pull requests listed here; the epic is done when the remaining datetime preimage optimizations are implemented and their associated work is closed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.