NVIDIA / NVIDIA/Model-Optimizer

Explicit INT8 Quantization Fails to Fuse Concat-Conv Block Compared to Implicit Mode

Open
#174 6 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

bug investigating model support needs trt/trtllm triage
Dominant language
Python
Stars
3.8k
Forks
604
Avg merge
2d 8h
Merged PRs (30d)
142

Description

I am currently trying to quantize my UNet-based pytorch model to INT8. I end up with a slightly different —and slower— engine graph compared to the one generated by implicit quantization.

More specifically, I’m struggling to properly quantize the Concat-Conv section of the network. I’ve tried all suggested Q/DQ placements from various issues (and more), but TensorRT never fully fuses the operators in that section.

Below, I’ll show a minimal example of the ONNX and TensorRT engine graphs I experimented with.

This is the fp32 ONNX model and the TRT engine generated by implicit quantization:

Image

Image

When i use the basic configuration of the quantization toolkit, which means i don't quantize concat (as proposed in this issue https://github.com/NVIDIA/TensorRT/issues/3861), i get these results:

Image

Image

The second thing tried was adding Q/DQ nodes to both inputs of concat (as proposed in this issue https://github.com/NVIDIA/TensorRT/issues/1144), this is the result:

Image

Image

Then i only quantized the residual path, which is my best try so far, however there are still unfused scale/pointwise operators:

Image

Image

If you have any idea how to correctly place the Q/DQ nodes in this scenario to match implicit quantization behavior, I’d really appreciate your insights. Also, if there’s a hidden or updated Q/DQ placement guide somewhere, I’d be very grateful if you could point me to it.
Thank you so much for your help and time!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.