deepspeedai / deepspeedai/DeepSpeed
[REQUEST] integrate powersgd to deepspeed
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Is your feature request related to a problem? Please describe.
yes. when the network bandwidth is limited, powersgd could be helpful
Describe the solution you'd like
some configuration to specify to enable PowerSGD when zero is larger than 0.
Describe alternatives you've considered
no.
Additional context
powersgd paper: https://arxiv.org/abs/1905.13727
used by openai's work of Dalle: https://arxiv.org/abs/2102.12092
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reading DeepSpeed's ZeRO configuration and the linked PowerSGD paper, then determine the integration points and configuration shape. Done means PowerSGD can be enabled through configuration for ZeRO when network bandwidth is limited.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100