deepspeedai / deepspeedai/DeepSpeedExamples

how to apply DeepSpeedExamples with mpirun ?

Open
#548 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6.8k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
1

Description

Hi i have been using mpi-operator, and found an interesting project according to https://github.com/dogukanutuna/mpi-operator/tree/master/examples/v2beta1/deepspeed which can apply cifar10_deepspeed.py with mpirun of mpi-operator on k8s cluster, and distributed training can be executed successfully for me.
So i wonder, can all of these script in DeepSpeedExamples/training be applyed by this mpi-operator-way with simple command mpirun ?
In my case, at least, gan_deepspeed_train.py can not be applied by such way with mpirun.
There were somethings about distributed training i found, however not sure whether they are correct or not:
When submitting command "deepspeed script.py --deepspeed". args.local_rank would be assigned implicitly according to my env before deepspeed.init_distributed(), I think it's critical for distributed training.
But when submitting "mpirun python script.py --deepspeed", args.local_rank can only be passed externally with manual parameter,
which goes against the convenience of mpi-operator.
any hint would be helpful to me. Thank you in advance~

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the DeepSpeedExamples/training scripts mentioned in the issue, especially gan_deepspeed_train.py, and compare their invocation with cifar10_deepspeed.py under mpi-operator. Determine whether the examples support mpirun and how local_rank is supplied; the issue does not define a specific code change or acceptance criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
cloud, distributed-systems, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.