apache / apache/datafusion-ballista
AQE: handle executor failure on the adaptive execution path
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge?**
The adaptive execution path (`AdaptiveExecutionGraph` / `AdaptivePlanner`) currently covers the happy path only and "does not cover executor failure". When AQE is enabled, the scheduler builds the stage DAG incrementally and re-optimizes the remaining plan as each shuffle stage completes. If an executor is lost, the runtime statistics and shuffle output that adaptive decisions were based on may no longer be available, and the AQE path has no recovery logic equivalent to the static `StaticExecutionGraph` path.
**Describe the solution you'd like**
Support executor-failure recovery on the AQE path: detect lost shuffle output for stages the adaptive planner has already resolved/consumed, re-run the affected stage(s), and re-drive adaptive re-optimization consistently after recovery.
**Additional context**
Related general failure-handling issues (not AQE-path specific): #1795, #1908.
## Context
This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in [`docs/source/user-guide/tuning-guide.md`](https://github.com/apache/datafusion-ballista/blob/main/docs/source/user-guide/tuning-guide.md) under "Current limitations" and in the doc comment on `AdaptiveExecutionGraph` (`ballista/scheduler/src/state/aqe/mod.rs`). Filing a discrete tracking issue so the gap is visible outside the doc.
Parent epic: #1359
Contributor guide
Research direction
Start by reading AdaptiveExecutionGraph and AdaptivePlanner in ballista/scheduler/src/state/aqe/mod.rs, then compare their failure handling with StaticExecutionGraph. Review the AQE limitation documented in docs/source/user-guide/tuning-guide.md. Done means lost shuffle output triggers affected-stage recovery and adaptive re-optimization continues consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100