kubeflow / kubeflow/mpi-operator
Work with DeepSpeed for large scale training
- Dominant language
- Go
- Stars
- 535
- Forks
- 238
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
[DeepSpeed](https://github.com/microsoft/DeepSpeed) is an excellent framework for training LLMs on a large scale, while the mpi-operator is the ideal tool to facilitate this within the Kubernetes ecosystem.
I'm planning to submit a series of PRs to make this project more ready-to-use for very large scale training with the DeepSpeed/mpi-style training framework.
The upcoming features may include the following modifications:
**Support for IP-style hostfile**
This is for performance efficiency and to prevent the environment variable length from exceeding its limit when using svc for those who wish to wrap it into an environment variable.
**Support for fault tolerance and elasticity**
This is a quasi-fault tolerance since NCCL communication must always be recreated when an error occurs. However, it's still worth implementing because recreating pods can be costly on a very large scale.
**Configuration decoupling**
There are some requirements that are currently left to the docker image maker to handle, such as _ssh_config_ and _sshd_config_. Perhaps the operator can manage all of these.
There are also some minor changes under consideration. Please feel free to share your thoughts on this topic.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by reviewing the mpi-operator's current hostfile, fault-tolerance and elasticity, and SSH configuration handling, then clarify the scope with maintainers; completion criteria are not specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- distributed-systems, infrastructure, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100