deepspeedai / deepspeedai/DeepSpeedExamples
How can I train step3 in DeepSpeed-Chat by pipeline parallelism?
@tjruwase is already working on this.
Since May 1, 2023.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
If I set "CUDA_VISIBLE_DEVICES=0,1,2,3" and then execute the script "run_6.7B.sh" in step3, then the model would be trained by data parallelism, and the data parallel world size is 4. There are no other parallelism(such as pipeline parallelism and tensor parallelism).
As a result, I can not train a big model if its size is larger than the capacity of a single GPU, even though I have many GPUs. Because without pipeline or tensor parallelism, I have to load a complete model into one GPU.
So, in DeepSpeed-Chat step3, what should I do if I want to train a big model which is too large to be loaded into one GPU? Maybe I have to rewrite the module class so it inherits from deepspeed.pipe.PipelineModule?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.