deepspeedai / deepspeedai/DeepSpeed
Enhancement: Clean up zero_to_fp32.py
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Current zero_to_fp32.py script is in need of optimization; currently the existing implementation requires 2X RAM for the parameters themselves.
Additionally, add documentation for when this script is necessary, how to plug into HuggingFace.
I'm happy to take this one, but cc'ing @stas00.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading zero_to_fp32.py and determine how its current parameter handling leads to the stated 2X RAM requirement. The work is done when the script's memory use is improved and documentation explains when it is needed and how to integrate it with HuggingFace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- documentation, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100