microsoft / microsoft/onnxruntime

[Feature Request] Let quantize_dynamic work with float16

Open
#15,341 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Currently quantize_dynamic() takes an onnx in float32 format an outputs an onnx in int8 format.

It would be nice so that this also worked on an onnx in float16 format.

Also, the new int8 onnx requires inputs of float32. While I would like to put intputs of float16 and it to work only with float16 internally.

### Describe scenario use case

Workflow speed up and possible inference speed up.
Currently my GPU can run float16 faster than float32. So if there is any float32 lurking inside the quantized model this is bad news.
This seems like a pretty easy fix to make. Just to add an options for float16.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating quantize_dynamic and tracing how float32 inputs become int8; no file or test is named in the issue. Done means float16 ONNX models can be processed and the resulting quantized model accepts float16 inputs without unwanted float32 computation, with coverage for both requested paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.