apache / apache/datafusion-ballista

Allow `ballista_scheduler` to be embedded in the process of other applications

Open
#568 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Recently, I am working with Ballista as a distributed query engine in our existing scheduling service systems.

I'm trying to embed `ballista_scheduler` into our already existing scheduling system due to the following cases:
* We have some internal private implementations of `TableProvider` that cannot be serialized into `ballista_scheduler` through the `ballista_client` using protobuf.
* Need to integrate `ballista_scheduler` and our existing scheduling system into one process.

I found that although `ballista_scheduler` has exposed the `scheduler_server` struct through the `pub mod scheduler_server`, but it still cannot be integrated and embedded.

Because the `state` and `submit_job` in the `scheduler_server` only are `pub(crate)`:
https://github.com/apache/arrow-ballista/blob/20891ae0a740c03b5a3a909ca033f45d59fcfc83/ballista/scheduler/src/scheduler_server/mod.rs#L59-L66
https://github.com/apache/arrow-ballista/blob/20891ae0a740c03b5a3a909ca033f45d59fcfc83/ballista/scheduler/src/scheduler_server/mod.rs#L150-L167

**Describe the solution you'd like**
Expose `state` and `submit_job` in the `scheduler_server` to pub.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in ballista/scheduler/src/scheduler_server/mod.rs at the SchedulerServer state and submit_job definitions referenced in the issue, then inspect their visibility and callers. Done means applications can access these components when embedding ballista_scheduler while existing scheduler behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.