oneapi-src / oneapi-src/oneAPI-samples

How to Perform Static Quantization Directly on an ONNX Model Using Intel® Neural Compressor?

Open
#2,488 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
C++
Stars
1.2k
Forks
745
PR merge metrics
No merged PRs in 30d

Description

Hello,

I'm using Intel® Neural Compressor (INC) to perform static quantization on my custom PyTorch model. I followed this script which demonstrates how to apply static quantization using INC on a PyTorch model.

My goal is to obtain the final quantized model in ONNX format. However, after quantization, saving the q_model results in a .pt file (PyTorch format). I also found that exporting quantized PyTorch models to ONNX is problematic due to limited support and compatibility issues, especially with static quantization.

My Question:

Is there a way to perform static quantization directly on an ONNX model using Intel® Neural Compressor to produce a quantized ONNX model as the output?
Alternatively, is there a specific method to export the statically quantized PyTorch model to ONNX format while addressing the compatibility issues?
Any guidance or examples on how to achieve this would be greatly appreciated.

Thank you!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked quantize_with_inc.ipynb sample and inspect how it saves the quantized PyTorch model. Verify whether the requested ONNX input/output path is supported and whether an export example belongs in this sample. Done means a clear, reproducible answer or example covering direct ONNX quantization and the alternative export path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.