deepspeedai / deepspeedai/DeepSpeed
Multi node multi gpu distributed load
Open
@tohtana is already working on this.
Since Jan 20, 2025.
enhancement
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hi
it seems we can load model in on one node then distribute model between node and train or inference but,
imagine we have 2 node each node 2 gpu with 24GB vram each gpu.
wanna loading model like gemma2 27B. one node can not load it. it need to distribute load between node from start till end without needing to load completely in one node.(imagine none of nodes can not offload for loading complete model on one node, like llama 3.1 405B).
is there a way to do this?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.