apache / apache/datafusion-ballista

AQE: coalesce and range-repartition-routing slots can co-exist on the same ExchangeExec

Open
#2,220 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

## Background

Every `ExchangeExec` in the AQE plan carries two independently-resolvable slots:

- **coalesce** (`CoalescePlan`): set by `CoalescePartitionsRule` based on runtime sub-part size heuristics — decides post-coalesce partition count K.
- **range-repartition routing** (`RangeRepartitionRouting`): set by `SchedulerAqe` when a range-repartitioned upstream stage completes and its cuts have been merged — carries `cuts` (length = K − 1).

## Bug

Both slots can land on the same `ExchangeExec`. `CoalescePartitionsRule::optimize` walks leaf exchanges and bails only on `broadcast`; it has no equivalent bail for range-repartitioned. `SchedulerAqe::repartition_routing` sets the routing slot regardless of what `CoalescePartitionsRule` has already committed.

When both are set, `BallistaAdapter::adapt_to_ballista`'s `(Some(cp), false)` arm builds a reader with `cp.groups.len()` partitions, then hands `cuts.len() + 1` predicates to `PerPartitionFilterExec::try_new` — which rejects the count mismatch.

Reference: https://github.com/apache/datafusion-ballista/pull/2196#discussion_r3705634917

## Proposed fix

Coalesce groups neighboring upstream partitions only, and range-re-partition cuts are monotonic in partition order, so contiguous-group coalescing is compatible with range partitioning: merge adjacent cut buckets into one when their upstream sub-parts get grouped.

Contributor guide

Open the contributing guide

Research direction

Start in BallistaAdapter::adapt_to_ballista to understand the partition-count mismatch, then trace CoalescePartitionsRule::optimize and SchedulerAqe::repartition_routing. Verify how contiguous coalesce groups interact with monotonic range cuts, and consider the issue done when coalesce and range-repartition routing can coexist without PerPartitionFilterExec rejecting their counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.