kubeflow / kubeflow/spark-operator

Support `managedBy` field on `SparkApplication` for integration with MultiKueue

Open
#2,879 3 comments 4 reactions 1 assignee Claimed by @1Ayush-Petwal View on GitHub
kind/feature
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What feature you would like to be added?

The support for the managedBy field which can delegate reconciliation from built-in controller, to a custom one.

The semantics of the field are: whenever the value is set, and it does not point to the built-in operator, then skip reconciliation

- the field is immutable
- the field is not defaulted

### Why is this needed?

Integration with [MultiKueue](https://kueue.sigs.k8s.io/docs/tasks/manage/setup_multikueue/) (multi-cluster Kueue):

- simpler installation (just follow the standard installation path), otherwise only installation of SparkApplication CRDs is required
- support for mixed setup in one cluster - some SparkApplication could be run by MultiKueue and some by the default operator

The `managedBy` field is already adopted by:

- https://github.com/kubernetes/enhancements/issues/4368 (beta in 1.32)
- https://github.com/kubernetes-sigs/jobset/issues/440 (released in 0.6)
- https://github.com/kubeflow/mpi-operator/issues/646 (released in 0.6)
- https://github.com/kubeflow/training-operator/issues/2193 (released in 1.9)
- https://github.com/ray-project/kuberay/issues/2544 (released in 1.3.0)

### Describe the solution you would like

Introduce `spec.managedBy` field in `SparkApplication`

### Describe alternatives you have considered

_No response_

### Additional context

as suggested in https://github.com/kubernetes-sigs/kueue/pull/7268#issuecomment-4063022268

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.