deepspeedai / deepspeedai/DeepSpeed
[BUG] MOE: Loading experts parameters error when using expert parallel.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
I have a model with 60 experts, and I am training the experts in parallel on two GPUs. Theoretically, GPU0 should load the parameters of the first 30 experts, while GPU1 should load the parameters of the last 30 experts. However, I found that both GPUs are loading the parameters of the first 30 experts . How should I modify this?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported setup with 60 experts and expert parallelism across two GPUs, then trace the expert-parameter loading path to determine why both ranks select the first 30 experts. Done means GPU0 loads experts 0–29 and GPU1 loads experts 30–59, with a regression test covering the two-GPU case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100