kubeflow / kubeflow/spark-operator

[FEATURE] Support Yunikorn as a batch scheduler

Open
#2,098 3 comments 6 reactions 0 assignees View on GitHub
enhancement lifecycle/frozen
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

#### What is the outcome that you are trying to reach?

[Apache YuniKorn](https://yunikorn.apache.org/) is an alternative Kubernetes scheduler that provides batch scheduling functionality, including gang scheduling, hierarchical queues and job ordering. The Spark operator should support Yunikorn as a batch scheduler by supplying the necessary task group information on the driver pod annotations.

#### Describe the solution you would like

- [x] Add a Yunikorn scheduler implementation that patches task group annotations and calculates `minResources` https://github.com/kubeflow/spark-operator/pull/2107
- [x] Allow a default batch scheduler to be specified so that all applications can be batch scheduled easily https://github.com/kubeflow/spark-operator/pull/2143
- [ ] Add a gang scheduling end to end test using Yunikorn to prevent future regressions
- [x] Prep work https://github.com/kubeflow/spark-operator/pull/2148
- [x] Add a doc page https://github.com/kubeflow/website/pull/3872

#### Describe alternatives you have considered

#### Additional context

Contributor guide

Open the contributing guide

Research direction

The remaining work is an end-to-end gang-scheduling test using YuniKorn; no source file or test path is named in the issue. Review the linked prep work in PR 2148 and the existing YuniKorn implementation, then add coverage that prevents regressions in gang scheduling.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, spark
Domain
distributed-systems, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.