microsoft / microsoft/onnxruntime
[Feature Request] Let quantize_dynamic work with float16
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Currently quantize_dynamic() takes an onnx in float32 format an outputs an onnx in int8 format.
It would be nice so that this also worked on an onnx in float16 format.
Also, the new int8 onnx requires inputs of float32. While I would like to put intputs of float16 and it to work only with float16 internally.
### Describe scenario use case
Workflow speed up and possible inference speed up.
Currently my GPU can run float16 faster than float32. So if there is any float32 lurking inside the quantized model this is bad news.
This seems like a pretty easy fix to make. Just to add an options for float16.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating quantize_dynamic and tracing how float32 inputs become int8; no file or test is named in the issue. Done means float16 ONNX models can be processed and the resulting quantized model accepts float16 inputs without unwanted float32 computation, with coverage for both requested paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100