How to Perform Static Quantization Directly on an ONNX Model Using Intel® Neural Compressor?

Open
#2,488 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Documentation
Clarity
Needs clarification
Activity status
Stale
Tech stack
python, pytorch
Domain
documentation, machine-learning

Research direction

Start with the linked quantize_with_inc.ipynb sample and inspect how it saves the quantized PyTorch model. Verify whether the requested ONNX input/output path is supported and whether an export example belongs in this sample. Done means a clear, reproducible answer or example covering direct ONNX quantization and the alternative export path.

Written by the indexing model from the issue text.

Description

question

Hello,

I'm using Intel® Neural Compressor (INC) to perform static quantization on my custom PyTorch model. I followed this script which demonstrates how to apply static quantization using INC on a PyTorch model.

My goal is to obtain the final quantized model in ONNX format. However, after quantization, saving the q_model results in a .pt file (PyTorch format). I also found that exporting quantized PyTorch models to ONNX is problematic due to limited support and compatibility issues, especially with static quantization.

My Question:

Is there a way to perform static quantization directly on an ONNX model using Intel® Neural Compressor to produce a quantized ONNX model as the output?
Alternatively, is there a specific method to export the statically quantized PyTorch model to ONNX format while addressing the compatibility issues?
Any guidance or examples on how to achieve this would be greatly appreciated.

Thank you!

Dominant language
C++
Stars
1.2k
Forks
745
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oneapi-src/oneAPI-samples

All issues in oneapi-src/oneAPI-samples

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.