deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Mixed dtype for model parameters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Is your feature request related to a problem? Please describe.
Is it possible to support model of irregular dtypes? For example, a large multimodal LLM might have a vision encoder that is of dtype=float32 and its LLM part in dtype=bfloat16. This will be particularly helpful since some customized vision models (e.g., MinkowskiEngine) don't support float16/bfloat16.
Describe the solution you'd like
Have a flag (e.g., dont_change_dtype) in DeepSpeedEngine to allow loading a nn.Module model without modifying its dtypes of various parameters (e.g., some params might be float32, while some are bfloat16)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the DeepSpeedEngine model-loading path and inspect how it changes the dtypes of parameters in a torch.nn.Module. The requested behavior is a flag that preserves mixed parameter dtypes, including float32 and bfloat16; done means models with irregular dtypes can load without those dtypes being modified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100