deepspeedai / deepspeedai/DeepSpeed
Support mixed precision mode O1 with zero optimization[REQUEST]
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Currently zero optimization mixed precision uses O2 mode (i.e. softmax is computed in fp16). This precision is usually unstable to train image generative models. Is there a plan to add O1 mixed precision support to zero optimization in the future?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named. Start by locating the zero optimization mixed-precision implementation and comparing its O2 behavior with the requested O1 mode. Done means O1 mixed precision is supported for zero optimization and its training stability is validated, but the issue provides no concrete acceptance test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100