apache / apache/datafusion-ballista

[EPIC] Add public APIs required for scheduler high availability

Open
#2,349 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Ballista exposes shared cluster state, job state, and job acquisition interfaces, but these interfaces are not sufficient for a library consumer to recover running jobs after a scheduler fails.

Completing recovery currently requires access to private scheduler state and execution graph internals. This issue tracks the public APIs required to implement scheduler high availability outside Ballista.

**Describe the solution you'd like**

Add public APIs for the following capabilities:

- [ ] **Execution graph persistence:** Provide versioned encoding and decoding for `ExecutionGraphBox`.
- [ ] **Recovered-job activation:** Allow a restored execution graph to be added to the active job set and made available for scheduling.
- [ ] **Ownership fencing:** Return an ownership token when a job is acquired and require it when saving job state, assigning tasks, and accepting task status. Updates from an earlier owner must be rejected.
- [ ] **Executor failover:** Allow executors to reconnect to another scheduler and retry task status updates that were not acknowledged.
- [ ] **Owner-aware control operations:** Allow status, cancellation, and cleanup requests to reach the scheduler that owns the job, or return enough information for the consumer to route them.

Completion criteria:

- An external crate can implement active-passive scheduler failover using only public Ballista APIs.
- A running query can continue after its scheduler process stops.
- Recovery is tested with adaptive query execution enabled and disabled.
- Updates from the previous scheduler owner are rejected.
- Task status updates are not lost during executor reconnection.
- Job status and cancellation work through the surviving scheduler.
- Incompatible execution graph versions produce a clear error.
- A public example demonstrates the complete recovery flow.

The following are outside the scope of this issue:

- A built-in consensus or membership system
- Selection of a durable state backend
- Shared shuffle or durable result storage
- Automatic high availability in the default scheduler binary

**Describe alternatives you've considered**

Consumers can maintain a Ballista fork or depend on private scheduler internals. Both approaches create version-specific integrations that cannot be supported as library APIs.

**Additional context**

This work is required before #2347 can document a complete scheduler recovery implementation.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Ballista’s existing public shared cluster-state, job-state, and job-acquisition interfaces, then inspect the ExecutionGraphBox persistence and scheduler recovery boundaries. Done means an external crate can implement active-passive failover using only public APIs, including version errors, ownership fencing, executor reconnection, recovery with adaptive query execution enabled and disabled, and surviving-scheduler status and cancellation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.