NVIDIA / NVIDIA/Model-Optimizer
Explicit INT8 Quantization Fails to Fuse Concat-Conv Block Compared to Implicit Mode
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 604
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 142
Description
I am currently trying to quantize my UNet-based pytorch model to INT8. I end up with a slightly different —and slower— engine graph compared to the one generated by implicit quantization.
More specifically, I’m struggling to properly quantize the Concat-Conv section of the network. I’ve tried all suggested Q/DQ placements from various issues (and more), but TensorRT never fully fuses the operators in that section.
Below, I’ll show a minimal example of the ONNX and TensorRT engine graphs I experimented with.
This is the fp32 ONNX model and the TRT engine generated by implicit quantization:
When i use the basic configuration of the quantization toolkit, which means i don't quantize concat (as proposed in this issue https://github.com/NVIDIA/TensorRT/issues/3861), i get these results:
The second thing tried was adding Q/DQ nodes to both inputs of concat (as proposed in this issue https://github.com/NVIDIA/TensorRT/issues/1144), this is the result:
Then i only quantized the residual path, which is my best try so far, however there are still unfused scale/pointwise operators:
If you have any idea how to correctly place the Q/DQ nodes in this scenario to match implicit quantization behavior, I’d really appreciate your insights. Also, if there’s a hidden or updated Q/DQ placement guide somewhere, I’d be very grateful if you could point me to it.
Thank you so much for your help and time!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.