Support functions date_bin and date_trunc in range partition satisfaction
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Child issue of EPIC https://github.com/apache/datafusion/issues/22395.
Consider this query
```
select a, binned
from my_range_partitioned_table
group by a, date_bin(1 hour, timestamp, 0) as binned
```
on a table which is `Range` partitioned on `timestamp` by day.
This query should not have a partial aggregate -> repartition -> final aggregate pipeline because
- 1 hour <= 1 day
- there are no incomplete intervals (ie. there's no 1 hour window which overlaps a split point; in other words, there's no 1 hour window that is partially in one partition and partially in another)
If the query were like this, then you cannot remove the repartition since 27 hours spans multiple partitions.
```
select a, binned
from my_range_partitioned_table
group by a, date_bin(27 hours, timestamp, 0) as binned
```
Due to subset satisfaction, both of these partitioning schemes should allow this query to avoid repartitions.
1. `Range` partitioned on `timestamp` by day
2. `Range` partitioned on `timestamp` by day and `a` by some arbitrary range.
We probably also want to include benchmarks for range partitioned queries at some point.
Related
- https://github.com/apache/datafusion/issues/23289
Contributor guide
Assessment
This issue has not been assessed yet.