apache / apache/datafusion

[EPIC] Implement Range Partitioning

Open
#22,395 26 comments 0 reactions 0 assignees View on GitHub
EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Problem

`Partitioning::Range` now exists as a physical partitioning model, but optimizer and planning code still need to preserve and use range metadata consistently. The goal of this epic is to make range-partitioned plans avoid unnecessary repartitioning while keeping multi-input correctness explicit.

Related discussion: #21992.

### Primary implementation order

- [x] #22397
- [x] #22589
- [x] #22778
- [x] #22645
- [x] #22786
- [x] #22787
- [x] #23137
- [x] #23236
- [x] #23191
- [x] #23183
- [x] #23290
- [x] #23289
- [x] #23451 , #23478 , #23479 (these were decided to be put into one PR)
- [x] #23452
- [x] #23453
- [x] #23454
- [x] #23266
- [x] #23450
- [x] #23376
- [x] #23455

### Optimizer and integration follow-ups

These are related but do not need to block the primary join and distribution-satisfaction path.

- [x] #23230
- [ ] #23436
- [x] #22394
- [ ] #22788
- [ ] #23569
- [x] #23572

### Documentation
- [ ] https://github.com/apache/datafusion/issues/24667

Contributor guide

Open the contributing guide

Research direction

Start with the related discussion #21992 and the remaining unchecked follow-ups #23436, #22788, #23569, and documentation issue #24667. Review the completed checklist for implementation context; done means addressing the selected follow-up scope and updating the epic's remaining checklist items.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.