deepspeedai / deepspeedai/DeepSpeed
[fp16] need to automatically convert fp32 inputs to fp16
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
While this hasn't been an issue with NLP models where inputs are int64 and embeddings convert those into the right float type based on model's weights dtype, for non-NLP models this conversion is lacking. And so the proposal is for Deepspeed to automatically convert float32 inputs to float16 if fp16 is enabled. And do nothing if the inputs aren't fp32.
While the user could convert it themselves this is not the norm since AMP or Apex do it automatically.
Context: https://github.com/huggingface/transformers/pull/11638#pullrequestreview-677894311
Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source files, tests, or entry points; begin by locating DeepSpeed's fp16 input handling and comparing the behavior described for AMP or Apex. Done means float32 inputs convert automatically when fp16 is enabled, while non-float32 inputs remain unchanged, with coverage for both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100