apache / apache/datafusion-ballista
[EPIC] Add public APIs required for scheduler high availability
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Ballista exposes shared cluster state, job state, and job acquisition interfaces, but these interfaces are not sufficient for a library consumer to recover running jobs after a scheduler fails.
Completing recovery currently requires access to private scheduler state and execution graph internals. This issue tracks the public APIs required to implement scheduler high availability outside Ballista.
**Describe the solution you'd like**
Add public APIs for the following capabilities:
- [ ] **Execution graph persistence:** Provide versioned encoding and decoding for `ExecutionGraphBox`.
- [ ] **Recovered-job activation:** Allow a restored execution graph to be added to the active job set and made available for scheduling.
- [ ] **Ownership fencing:** Return an ownership token when a job is acquired and require it when saving job state, assigning tasks, and accepting task status. Updates from an earlier owner must be rejected.
- [ ] **Executor failover:** Allow executors to reconnect to another scheduler and retry task status updates that were not acknowledged.
- [ ] **Owner-aware control operations:** Allow status, cancellation, and cleanup requests to reach the scheduler that owns the job, or return enough information for the consumer to route them.
Completion criteria:
- An external crate can implement active-passive scheduler failover using only public Ballista APIs.
- A running query can continue after its scheduler process stops.
- Recovery is tested with adaptive query execution enabled and disabled.
- Updates from the previous scheduler owner are rejected.
- Task status updates are not lost during executor reconnection.
- Job status and cancellation work through the surviving scheduler.
- Incompatible execution graph versions produce a clear error.
- A public example demonstrates the complete recovery flow.
The following are outside the scope of this issue:
- A built-in consensus or membership system
- Selection of a durable state backend
- Shared shuffle or durable result storage
- Automatic high availability in the default scheduler binary
**Describe alternatives you've considered**
Consumers can maintain a Ballista fork or depend on private scheduler internals. Both approaches create version-specific integrations that cannot be supported as library APIs.
**Additional context**
This work is required before #2347 can document a complete scheduler recovery implementation.
Contributor guide
Research direction
Start by tracing Ballista’s existing public shared cluster-state, job-state, and job-acquisition interfaces, then inspect the ExecutionGraphBox persistence and scheduler recovery boundaries. Done means an external crate can implement active-passive failover using only public APIs, including version errors, ownership fencing, executor reconnection, recovery with adaptive query execution enabled and disabled, and surviving-scheduler status and cancellation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100