deepspeedai / deepspeedai/DeepSpeed
How to save DS Zero model weights in meta tensor format for DS Inference
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hi there, I'm wondering if there's a way to save the weights of a model that's been trained using DS Zero directly into the meta tensor format that is used to speed up model loading times for DS Inference.
Currently, I've been saving the model back into a single .bin file and then loading that into DS Inference & then saving the meta tensor out, but wondering if there is a quicker way to do this.
Any help would be greatly appreciated!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue concerns saving DeepSpeed ZeRO-trained weights directly in the meta tensor format used by DS Inference. Start by tracing the existing ZeRO save path and DS Inference loading path, then determine whether a direct conversion path exists; done means the supported workflow is implemented or documented and avoids the intermediate single .bin step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100