apache / apache/datafusion-ballista

AQE: switch from streaming aggregation to hash aggregation based on runtime statistics

Open
#1,989 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge?**

The adaptive execution path does not switch from streaming aggregation to hash aggregation (or vice versa) based on runtime statistics. Streaming aggregation depends on sorted input and assumptions about group cardinality; when runtime statistics contradict the plan-time estimate, the chosen aggregation strategy can be suboptimal.

**Describe the solution you'd like**

Add an adaptive physical optimizer rule that reconsiders the aggregation strategy for not-yet-resolved aggregate stages using exact runtime statistics (input ordering, group cardinality), switching between streaming and hash aggregation as appropriate.

## Context

This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in [`docs/source/user-guide/tuning-guide.md`](https://github.com/apache/datafusion-ballista/blob/main/docs/source/user-guide/tuning-guide.md) under "Current limitations" and in the doc comment on `AdaptiveExecutionGraph` (`ballista/scheduler/src/state/aqe/mod.rs`). Filing a discrete tracking issue so the gap is visible outside the doc.

Parent epic: #1359

Contributor guide

Open the contributing guide

Research direction

Start with the AQE limitation in docs/source/user-guide/tuning-guide.md and the AdaptiveExecutionGraph documentation in ballista/scheduler/src/state/aqe/mod.rs. Trace how unresolved aggregate stages and runtime statistics are handled, then define completion as an adaptive rule that switches between streaming and hash aggregation using input ordering and group cardinality.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.