apache / apache/datafusion-ballista

Job hangs indefinitely instead of failing when all executors are lost

Open
#2,029 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

## Describe the bug

If every executor dies while a query is running, the job **never terminates**. The scheduler waits indefinitely rather than failing the query, so the client hangs forever.

## To Reproduce

Reproduced end to end on a real multi-process cluster by the chaos harness in #2026 (scenario `killing_every_executor_terminates_the_job`, which fails under **both** AQE on and AQE off):

1. Start a scheduler and 2 executors.
2. Submit a query (the harness slows a stage with a delay UDF so the kill lands while tasks are genuinely running).
3. Wait until stage 1 has running tasks.
4. `SIGKILL` **both** executors.
5. Wait.

The job does not terminate within 120 seconds. The harness asserts only that the job *ends* — it makes no claim about success — and even that does not hold.

## Expected behavior

The job should fail with a clear error once the scheduler determines its tasks cannot be scheduled — there are no executors left to run them, and no prospect of recovery without a new executor registering.

Some judgement is needed on the exact policy, and reasonable people could disagree:

- Failing fast once all executors are lost is the simplest behavior and is what a client most likely wants.
- Alternatively, if the intent is to *wait* for executors to come back (a plausible design for a long-lived cluster), then that wait should be **bounded** by a configurable timeout, after which the job fails — and ideally the job should be visibly reported as blocked-on-resources in the meantime rather than looking indistinguishable from a job that is simply slow.

Either way, an unbounded silent hang is not a good outcome: the caller has no signal and no way to distinguish it from progress.

## Additional context

Found while building an HA chaos-testing harness (#2026) that drives the scheduler's fault-tolerance paths on a real multi-process cluster. This scenario is deliberately a **hang detector** rather than a correctness test.

Lower severity than the two sibling issues (the `FetchFailed` and `Shared`-wrapping misclassifications), since total executor loss is a more extreme condition than a single executor dying — but the failure mode (silent unbounded hang) is worse than a clean error.

Contributor guide

Open the contributing guide

Research direction

Start with the chaos harness scenario `killing_every_executor_terminates_the_job` from #2026 and trace the scheduler's fault-tolerance path when both executors are killed. Decide whether executor loss should fail immediately or use a bounded wait, then make the scenario finish with a clear job failure instead of hanging indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.