deepspeedai / deepspeedai/DeepSpeedExamples
The problem of model parallelism in training
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
In most cases, I think we need model parallelism more than data parallelism. It is hoped that the model can be trained in parallel, because the current model is very large, 80G graphics card is not enough to load a complete actor model, reference model, critic model and reward model. For example, I used llama 7B as the actor model and reference model, critic model and reward model, and the model could not be fully loaded on the three A100 80G, and OOM appeared.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reviewing the repository's training examples and clarify the intended model-parallelism scope, supported models, and hardware assumptions. Done would require an agreed implementation or example that can load and train the stated model components without the reported out-of-memory failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100