apache / apache/datafusion-ballista
Allow `ballista_scheduler` to be embedded in the process of other applications
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Recently, I am working with Ballista as a distributed query engine in our existing scheduling service systems.
I'm trying to embed `ballista_scheduler` into our already existing scheduling system due to the following cases:
* We have some internal private implementations of `TableProvider` that cannot be serialized into `ballista_scheduler` through the `ballista_client` using protobuf.
* Need to integrate `ballista_scheduler` and our existing scheduling system into one process.
I found that although `ballista_scheduler` has exposed the `scheduler_server` struct through the `pub mod scheduler_server`, but it still cannot be integrated and embedded.
Because the `state` and `submit_job` in the `scheduler_server` only are `pub(crate)`:
https://github.com/apache/arrow-ballista/blob/20891ae0a740c03b5a3a909ca033f45d59fcfc83/ballista/scheduler/src/scheduler_server/mod.rs#L59-L66
https://github.com/apache/arrow-ballista/blob/20891ae0a740c03b5a3a909ca033f45d59fcfc83/ballista/scheduler/src/scheduler_server/mod.rs#L150-L167
**Describe the solution you'd like**
Expose `state` and `submit_job` in the `scheduler_server` to pub.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start in ballista/scheduler/src/scheduler_server/mod.rs at the SchedulerServer state and submit_job definitions referenced in the issue, then inspect their visibility and callers. Done means applications can access these components when embedding ballista_scheduler while existing scheduler behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100