deepspeedai / deepspeedai/DeepSpeed
Support zeroth order backpropagation (300X faster training)
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
ZORB is a paradigm shift that enable ~300X faster training while reducing not dramatically throughout hence it has value and would gain to be more known.
Hence why integrating it as an optional optimization pass for Deepspeed would make sense.
https://arxiv.org/abs/2011.08895
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reading the linked ZORB paper and reviewing DeepSpeed's existing optimization-pass architecture. Done should mean ZORB is available as an optional DeepSpeed optimization and its training-speed and output-quality effects are demonstrated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100