kubeflow / kubeflow/mpi-operator
Replace the plain pod workers with Indexed Job
- Dominant language
- Go
- Stars
- 535
- Forks
- 238
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
Part-of: #373
Currently, the mpi-operator manages the plain pod workers. However, the management mechanism is similar to kubernetes batch/job, which is a reinvention of the wheel, although I understand the batch/job didn't have all features to replace the plain pod with batch/job in the past.
Because the Indexed job supports Elastically ([Elastic Indexed job](https://kubernetes.io/docs/concepts/workloads/controllers/job/#elastic-indexed-jobs)) by default since the kubernetes v1.27, even if we replace the plain pod management with Indexed job, we can support MPIJob with elastic semantics like the horovod.
So, I would propose replacing the plain pod workers with Indexd Job after the kubernetes v1.26 (EoL: 2024-02-28) has been deprecated.
Let me know what you think. @alculquicondor @terrytangyuan
Contributor guide
Research direction
Start with the parent issue #373 and the linked Kubernetes Indexed Job and Elastic Indexed Job documentation. Establish the migration scope, including the Kubernetes version boundary and how MPIJob elastic semantics should be preserved; the work is done when the replacement design and compatibility expectations are settled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100