Implement a way to preserve partitioning through `UnionExec` without losing ordering
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
The `EnforceDistribution` physical optimizer pass in DataFusion in some cases will introduce `InterleaveExec` to increase partitioning when data passes through a `UnionExec`:
https://github.com/apache/datafusion/blob/22311835bc1b4bd83b50e1c3875b0e725622b872/datafusion/core/src/physical_optimizer/enforce_distribution.rs#L1196-L1226
Here is what `InterleaveExec` does: https://github.com/apache/datafusion/blob/4edbdd7d09d97f361748c086afbd7b3dda972f76/datafusion/physical-plan/src/union.rs#L286-L317
However, this has the potential downside of destroying and pre-existing ordering which is sometimes preferable than increasing / improving partitionining (e.g. see https://github.com/apache/datafusion/issues/10257 and `datafusion.optimizer.prefer_existing_sort` setting)
### Describe the solution you'd like
I would like there to be some way to preserve the partitioning after a `UnionExec` without losing the ordering information and then remove the `prefer_existing_union` flag
### Describe alternatives you've considered
One possibility is to add a `preserve_order` flag to `InterleaveExec` the same way as `RepartitionExec` has a `preserve_order` flag: https://github.com/apache/datafusion/blob/4edbdd7d09d97f361748c086afbd7b3dda972f76/datafusion/physical-plan/src/repartition/mod.rs#L328-L417
### Additional context
We encountered this while working on https://github.com/apache/datafusion/pull/10259 @mustafasrepo and @phillipleblanc pointed out that config flag `prefer_existing_union` was effectively the same as `prefer_existing_sort`
Contributor guide
Research direction
Start with the EnforceDistribution logic in datafusion/core/src/physical_optimizer/enforce_distribution.rs and the InterleaveExec implementation in datafusion/physical-plan/src/union.rs; compare their behavior with RepartitionExec in datafusion/physical-plan/src/repartition/mod.rs. Determine how UnionExec can retain existing ordering while preserving partitioning, then verify that the redundant prefer_existing_union flag can be removed without losing the prefer_existing_sort behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100