apache / apache/datafusion-ballista
Job hangs indefinitely instead of failing when all executors are lost
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
## Describe the bug
If every executor dies while a query is running, the job **never terminates**. The scheduler waits indefinitely rather than failing the query, so the client hangs forever.
## To Reproduce
Reproduced end to end on a real multi-process cluster by the chaos harness in #2026 (scenario `killing_every_executor_terminates_the_job`, which fails under **both** AQE on and AQE off):
1. Start a scheduler and 2 executors.
2. Submit a query (the harness slows a stage with a delay UDF so the kill lands while tasks are genuinely running).
3. Wait until stage 1 has running tasks.
4. `SIGKILL` **both** executors.
5. Wait.
The job does not terminate within 120 seconds. The harness asserts only that the job *ends* — it makes no claim about success — and even that does not hold.
## Expected behavior
The job should fail with a clear error once the scheduler determines its tasks cannot be scheduled — there are no executors left to run them, and no prospect of recovery without a new executor registering.
Some judgement is needed on the exact policy, and reasonable people could disagree:
- Failing fast once all executors are lost is the simplest behavior and is what a client most likely wants.
- Alternatively, if the intent is to *wait* for executors to come back (a plausible design for a long-lived cluster), then that wait should be **bounded** by a configurable timeout, after which the job fails — and ideally the job should be visibly reported as blocked-on-resources in the meantime rather than looking indistinguishable from a job that is simply slow.
Either way, an unbounded silent hang is not a good outcome: the caller has no signal and no way to distinguish it from progress.
## Additional context
Found while building an HA chaos-testing harness (#2026) that drives the scheduler's fault-tolerance paths on a real multi-process cluster. This scenario is deliberately a **hang detector** rather than a correctness test.
Lower severity than the two sibling issues (the `FetchFailed` and `Shared`-wrapping misclassifications), since total executor loss is a more extreme condition than a single executor dying — but the failure mode (silent unbounded hang) is worse than a clean error.
Contributor guide
Research direction
Start with the chaos harness scenario `killing_every_executor_terminates_the_job` from #2026 and trace the scheduler's fault-tolerance path when both executors are killed. Decide whether executor loss should fail immediately or use a bounded wait, then make the scenario finish with a clear job failure instead of hanging indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100