apache / apache/datafusion

Implement a way to preserve partitioning through `UnionExec` without losing ordering

Open
#10,314 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The `EnforceDistribution` physical optimizer pass in DataFusion in some cases will introduce `InterleaveExec` to increase partitioning when data passes through a `UnionExec`:

https://github.com/apache/datafusion/blob/22311835bc1b4bd83b50e1c3875b0e725622b872/datafusion/core/src/physical_optimizer/enforce_distribution.rs#L1196-L1226

Here is what `InterleaveExec` does: https://github.com/apache/datafusion/blob/4edbdd7d09d97f361748c086afbd7b3dda972f76/datafusion/physical-plan/src/union.rs#L286-L317

However, this has the potential downside of destroying and pre-existing ordering which is sometimes preferable than increasing / improving partitionining (e.g. see https://github.com/apache/datafusion/issues/10257 and `datafusion.optimizer.prefer_existing_sort` setting)

### Describe the solution you'd like

I would like there to be some way to preserve the partitioning after a `UnionExec` without losing the ordering information and then remove the `prefer_existing_union` flag

### Describe alternatives you've considered

One possibility is to add a `preserve_order` flag to `InterleaveExec` the same way as `RepartitionExec` has a `preserve_order` flag: https://github.com/apache/datafusion/blob/4edbdd7d09d97f361748c086afbd7b3dda972f76/datafusion/physical-plan/src/repartition/mod.rs#L328-L417

### Additional context

We encountered this while working on https://github.com/apache/datafusion/pull/10259 @mustafasrepo and @phillipleblanc pointed out that config flag `prefer_existing_union` was effectively the same as `prefer_existing_sort`

Contributor guide

Open the contributing guide

Research direction

Start with the EnforceDistribution logic in datafusion/core/src/physical_optimizer/enforce_distribution.rs and the InterleaveExec implementation in datafusion/physical-plan/src/union.rs; compare their behavior with RepartitionExec in datafusion/physical-plan/src/repartition/mod.rs. Determine how UnionExec can retain existing ordering while preserving partitioning, then verify that the redundant prefer_existing_union flag can be removed without losing the prefer_existing_sort behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.