apache / apache/datafusion-ballista

AQE: switch from hash join to sort-merge join based on runtime statistics

Open
#1,988 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge?**

The adaptive execution path does not switch a `HashJoinExec` to a `SortMergeJoinExec` (or vice versa) based on runtime statistics collected from completed stages. When exact row counts / byte sizes reveal that a join input is far larger than estimated at plan time, keeping a hash join can cause excessive memory pressure or spilling where a sort-merge join would be more robust.

**Describe the solution you'd like**

Add an adaptive physical optimizer rule that reconsiders the join algorithm for not-yet-resolved join stages using exact runtime statistics, switching between hash join and sort-merge join as appropriate.

**Additional context**

Closed issue #1648 discussed whether Ballista should default to sort-merge join generally; this issue is specifically about a *runtime* AQE switch driven by collected statistics.

## Context

This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in [`docs/source/user-guide/tuning-guide.md`](https://github.com/apache/datafusion-ballista/blob/main/docs/source/user-guide/tuning-guide.md) under "Current limitations" and in the doc comment on `AdaptiveExecutionGraph` (`ballista/scheduler/src/state/aqe/mod.rs`). Filing a discrete tracking issue so the gap is visible outside the doc.

Parent epic: #1359

Contributor guide

Open the contributing guide

Research direction

Start with the AdaptiveExecutionGraph doc comment in ballista/scheduler/src/state/aqe/mod.rs and the AQE limitation in docs/source/user-guide/tuning-guide.md. Trace how runtime statistics reach unresolved join stages and define the switching rule for HashJoinExec and SortMergeJoinExec. Done means runtime row counts and byte sizes can select either join strategy, with coverage for both directions.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.