apache / apache/datafusion-ballista

AQE: handle executor failure on the adaptive execution path

Open
#1,986 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge?**

The adaptive execution path (`AdaptiveExecutionGraph` / `AdaptivePlanner`) currently covers the happy path only and "does not cover executor failure". When AQE is enabled, the scheduler builds the stage DAG incrementally and re-optimizes the remaining plan as each shuffle stage completes. If an executor is lost, the runtime statistics and shuffle output that adaptive decisions were based on may no longer be available, and the AQE path has no recovery logic equivalent to the static `StaticExecutionGraph` path.

**Describe the solution you'd like**

Support executor-failure recovery on the AQE path: detect lost shuffle output for stages the adaptive planner has already resolved/consumed, re-run the affected stage(s), and re-drive adaptive re-optimization consistently after recovery.

**Additional context**

Related general failure-handling issues (not AQE-path specific): #1795, #1908.

## Context

This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in [`docs/source/user-guide/tuning-guide.md`](https://github.com/apache/datafusion-ballista/blob/main/docs/source/user-guide/tuning-guide.md) under "Current limitations" and in the doc comment on `AdaptiveExecutionGraph` (`ballista/scheduler/src/state/aqe/mod.rs`). Filing a discrete tracking issue so the gap is visible outside the doc.

Parent epic: #1359

Contributor guide

Open the contributing guide

Research direction

Start by reading AdaptiveExecutionGraph and AdaptivePlanner in ballista/scheduler/src/state/aqe/mod.rs, then compare their failure handling with StaticExecutionGraph. Review the AQE limitation documented in docs/source/user-guide/tuning-guide.md. Done means lost shuffle output triggers affected-stage recovery and adaptive re-optimization continues consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.