microsoft / microsoft/onnxruntime-genai

Support for dynamic int8 quantization

Open
#1,792 0 comments 0 reactions 0 assignees View on GitHub
quantization
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

Hi there!

Could we add the functionality to add int8 quantization as a `--precision` option?

I believe the easiest way would probably be using onnxruntime `quantize_dynamic` function after converting to fp32. Not really sure what the difference would be for using MatMulNBitsQuantizer with `bits=8` option.

The reason is that some architecutres (specifically ppc64le) use matrix math optimization for int8 but do not have them for int4.

Would a PR in that direction be accepted?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing --precision handling and the path that converts models to fp32. Compare the proposed onnxruntime quantize_dynamic approach with MatMulNBitsQuantizer using bits=8. Done means int8 dynamic quantization is available through --precision and supports the ppc64le use case; no file or test is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.