apache / apache/datafusion

Support functions date_bin and date_trunc in range partition satisfaction

Open
#23,569 8 comments 3 reactions 1 assignee Assigned to @benbellick View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Child issue of EPIC https://github.com/apache/datafusion/issues/22395.

Consider this query
```
select a, binned
from my_range_partitioned_table
group by a, date_bin(1 hour, timestamp, 0) as binned
```
on a table which is `Range` partitioned on `timestamp` by day.

This query should not have a partial aggregate -> repartition -> final aggregate pipeline because
- 1 hour <= 1 day
- there are no incomplete intervals (ie. there's no 1 hour window which overlaps a split point; in other words, there's no 1 hour window that is partially in one partition and partially in another)

If the query were like this, then you cannot remove the repartition since 27 hours spans multiple partitions.
```
select a, binned
from my_range_partitioned_table
group by a, date_bin(27 hours, timestamp, 0) as binned
```

Due to subset satisfaction, both of these partitioning schemes should allow this query to avoid repartitions.
1. `Range` partitioned on `timestamp` by day
2. `Range` partitioned on `timestamp` by day and `a` by some arbitrary range.

We probably also want to include benchmarks for range partitioned queries at some point.

Related
- https://github.com/apache/datafusion/issues/23289

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.