apache / apache/datafusion-ballista

Design note: composing multi-partition tasks with plan-level AQE in DataFusion

Open
#2,445 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge?**

Multi-partition tasks widen the plan that a single coordinator can see inside
one executor. That opens a design question we have no home for: how Ballista's
cluster-level AQE should compose with the plan-level adaptivity DataFusion is
exploring.

This was written up as a "Composition with in-flight DataFusion AQE" subsection
in the contributors guide. It was removed in #2438 because it anchored a guide
section to two upstream PoCs that have not landed:

- apache/datafusion#23026 is still an open draft, last touched 2026-08-24
- apache/datafusion#23167 was closed without merging

Filing it here so the thinking survives without the guide claiming it describes
current behaviour.

**Describe the solution you'd like**

The removed text, verbatim:

> Two upstream DataFusion PoCs are converging on the same primitives at the
> single-plan level: [apache/datafusion#23026](https://github.com/apache/datafusion/pull/23026)
> adds `RangeRepartitionExec`, `HaloDropExec`, and a `runtime_partition_extremes`
> trait method to parallelize `RANGE`-frame windows inside one plan;
> [apache/datafusion#23167](https://github.com/apache/datafusion/pull/23167) adds
> `PipelineBreakerBuffer` + `RuntimeOptimizerExec` + a `RuntimeRule` trait so a
> plan-root coordinator can observe post-pipeline-breaker runtime stats and
> mutate adaptive operators — build-side swaps, split points, skew fixes — in
> place, streaming-native, no disk materialization. Multi-partition tasks widens
> the plan a single coordinator sees: one `RuntimeOptimizerExec` now observes the
> full slice's pipeline-breaker state, so `RangeRepartitionExec`'s halo-aware
> routing and any `RuntimeRule`'s adaptive decisions cover an executor's whole
> vcore budget instead of one core. Ballista's AQE stage barriers are the
> cluster-scale analog of that plan-root coordinator, and the same rule library
> lifts unchanged: sketches and row counts collected inside each slice-task get
> reported at the shuffle boundary, and the scheduler applies the same rules
> cluster-wide. Three levels, one rule library — intra-plan (DataFusion),
> intra-executor slice (multi-partition tasks), inter-executor stage boundary
> (Ballista AQE).

**Describe alternatives you've considered**

Keeping it in the contributors guide. That is what #2438 moved away from: the
guide should describe code we have, and neither upstream PR is merged, so the
section could not be kept accurate.

**Additional context**

Ballista already carries one piece of this. `PartitionedBoundedWindowAggExec`
wraps DataFusion's `BoundedWindowAggExec` to override its `SinglePartition`
requirement, and its rustdoc cites apache/datafusion#23026 as the change that
would let the wrapper collapse. That link is deliberate and stays: it explains
why a workaround exists, rather than describing behaviour we do not have.

This issue should be closed if the upstream direction is abandoned, or folded
into the AQE epic (#1359) if it is picked up.

Contributor guide

Open the contributing guide

Research direction

Start with the existing PartitionedBoundedWindowAggExec wrapper and its rustdoc reference to apache/datafusion#23026, then review AQE epic #1359 and the status of both upstream PoCs. This is complete only if the upstream direction is abandoned and the issue closes, or if the design is folded into the AQE epic when the work is picked up.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.