apache / apache/datafusion-ballista
AQE: coalesce and range-repartition-routing slots can co-exist on the same ExchangeExec
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
## Background
Every `ExchangeExec` in the AQE plan carries two independently-resolvable slots:
- **coalesce** (`CoalescePlan`): set by `CoalescePartitionsRule` based on runtime sub-part size heuristics — decides post-coalesce partition count K.
- **range-repartition routing** (`RangeRepartitionRouting`): set by `SchedulerAqe` when a range-repartitioned upstream stage completes and its cuts have been merged — carries `cuts` (length = K − 1).
## Bug
Both slots can land on the same `ExchangeExec`. `CoalescePartitionsRule::optimize` walks leaf exchanges and bails only on `broadcast`; it has no equivalent bail for range-repartitioned. `SchedulerAqe::repartition_routing` sets the routing slot regardless of what `CoalescePartitionsRule` has already committed.
When both are set, `BallistaAdapter::adapt_to_ballista`'s `(Some(cp), false)` arm builds a reader with `cp.groups.len()` partitions, then hands `cuts.len() + 1` predicates to `PerPartitionFilterExec::try_new` — which rejects the count mismatch.
Reference: https://github.com/apache/datafusion-ballista/pull/2196#discussion_r3705634917
## Proposed fix
Coalesce groups neighboring upstream partitions only, and range-re-partition cuts are monotonic in partition order, so contiguous-group coalescing is compatible with range partitioning: merge adjacent cut buckets into one when their upstream sub-parts get grouped.
Contributor guide
Research direction
Start in BallistaAdapter::adapt_to_ballista to understand the partition-count mismatch, then trace CoalescePartitionsRule::optimize and SchedulerAqe::repartition_routing. Verify how contiguous coalesce groups interact with monotonic range cuts, and consider the issue done when coalesce and range-repartition routing can coexist without PerPartitionFilterExec rejecting their counts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100