deepspeedai / deepspeedai/DeepSpeedExamples
how to apply DeepSpeedExamples with mpirun ?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
Hi i have been using mpi-operator, and found an interesting project according to https://github.com/dogukanutuna/mpi-operator/tree/master/examples/v2beta1/deepspeed which can apply cifar10_deepspeed.py with mpirun of mpi-operator on k8s cluster, and distributed training can be executed successfully for me.
So i wonder, can all of these script in DeepSpeedExamples/training be applyed by this mpi-operator-way with simple command mpirun ?
In my case, at least, gan_deepspeed_train.py can not be applied by such way with mpirun.
There were somethings about distributed training i found, however not sure whether they are correct or not:
When submitting command "deepspeed script.py --deepspeed". args.local_rank would be assigned implicitly according to my env before deepspeed.init_distributed(), I think it's critical for distributed training.
But when submitting "mpirun python script.py --deepspeed", args.local_rank can only be passed externally with manual parameter,
which goes against the convenience of mpi-operator.
any hint would be helpful to me. Thank you in advance~
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the DeepSpeedExamples/training scripts mentioned in the issue, especially gan_deepspeed_train.py, and compare their invocation with cifar10_deepspeed.py under mpi-operator. Determine whether the examples support mpirun and how local_rank is supplied; the issue does not define a specific code change or acceptance criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- cloud, distributed-systems, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100