NVIDIA / NVIDIA/Model-Optimizer
INT4 Quantization Export
Open
Nobody has claimed this yet.
bug
torch.quantization
triaged
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 604
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 142
Description
Before submitting an issue, please make sure it hasn't been already addressed by searching through the existing and past issues.
Describe the bug
Hi, I encountered an issue. I have tried quantizing the vit_base_patch16_224 model in INT4 . The problem is that when exporting to ONNX, the pre_process function in INT4QuantExporter can only match subgraphs like DequantizeLinear -> Reshape -> Transpose -> MatMul/Gemm. In fact, there are other subgraphs, such as DequantizeLinear -> Transpose -> MatMul/Gemm, which cannot be handled successfully. How can I fix this? Thanks.
- ?
Steps/Code to reproduce bug
- ?
Expected behavior
Who can help?
- ?
System information
- Container used (if applicable): ?
- OS (e.g., Ubuntu 22.04, CentOS 7, Windows 10): ?
- CPU architecture (x86_64, aarch64): ?
- GPU name (e.g. H100, A100, L40S): ?
- GPU memory size: ?
- Number of GPUs: ?
- Library versions (if applicable):
- Python: ?
- ModelOpt version or commit hash: ?
- CUDA: ?
- PyTorch: ?
- Transformers: ?
- TensorRT-LLM: ?
- ONNXRuntime: ?
- TensorRT: ?
- Any other details that may help: ?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.