deepspeedai / deepspeedai/DeepSpeedExamples
How to use data parallelism between nodes?
Open
@jeffra is already working on this.
Since May 1, 2023.
auto-feature-selection
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
Suppose there are more than one compute nodes, and each node's GPU memory in total is enough to contain a model. What is the recommend way to configure DeepSpeed if I want to use zero-3 within each node and use data parallel between these nodes? Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.