kubeflow / kubeflow/mpi-operator

Work with DeepSpeed for large scale training

Open
#611 28 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

[DeepSpeed](https://github.com/microsoft/DeepSpeed) is an excellent framework for training LLMs on a large scale, while the mpi-operator is the ideal tool to facilitate this within the Kubernetes ecosystem.

I'm planning to submit a series of PRs to make this project more ready-to-use for very large scale training with the DeepSpeed/mpi-style training framework.

The upcoming features may include the following modifications:

**Support for IP-style hostfile**
This is for performance efficiency and to prevent the environment variable length from exceeding its limit when using svc for those who wish to wrap it into an environment variable.

**Support for fault tolerance and elasticity**
This is a quasi-fault tolerance since NCCL communication must always be recreated when an error occurs. However, it's still worth implementing because recreating pods can be costly on a very large scale.

**Configuration decoupling**
There are some requirements that are currently left to the docker image maker to handle, such as _ssh_config_ and _sshd_config_. Perhaps the operator can manage all of these.

There are also some minor changes under consideration. Please feel free to share your thoughts on this topic.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by reviewing the mpi-operator's current hostfile, fault-tolerance and elasticity, and SSH configuration handling, then clarify the scope with maintainers; completion criteria are not specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
distributed-systems, infrastructure, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.