deepspeedai / deepspeedai/DeepSpeed
How to use both deepspeed framework and tutel framework?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hello，I use both your deepspeed and tutel framework, 4 experts and 4 gpus, both Data and Expert Parallelism.
To avoid expert parameters be managed by deepspeed allreduce, code :
deepspeed.initialize( args=self.args,
model=model,
model_parameters = [param for name,param in model.named_parameters() if not hasattr(param, "skip_allruduce")],
config=ds_config )
but it doesn't work，expert parameters cann't be updated correctly. how should I do ? Thanks!
If I use both DDP and tutel framework, it's ok like this:
https://github.com/microsoft/tutel/issues/204#issuecomment-1548182375
How to use deepspeed correctly like DDP?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the four-expert, four-GPU setup around deepspeed.initialize and the model_parameters filter shown in the issue, then compare it with the linked Tutel DDP example. Done means expert parameters update correctly when DeepSpeed and Tutel are used together with data and expert parallelism.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100