Lower-than-Expected Performance Improvement with INT8 Quantization in TensorRT 10.0 on A100 GPU

Description

I recently attempted to utilize INT8 quantization with Stable Diffusion XL to enhance inference performance based on the claims made in a recent TensorRT blog post, which suggested that this approach could achieve a performance improvement of nearly 2x. However, my experiences do not align with these expectations. After implementing INT8 quantization, the performance improvement was notably less than advertised.

Environment

TensorRT Version:
10.0.0b6

NVIDIA GPU:
A100

Operating System:
Python Version:
3.10

Baremetal or Container (if so, version):
Triton 24.03

Relevant Files

Logs:
https://yaso.su/SDXLTestLogs

Steps To Reproduce

I closely followed the steps laid out in the README of the TensorRT repository for the Stable Diffusion XL demo, which you can find here: https://github.com/NVIDIA/TensorRT/tree/release/10.0/demo/Diffusion. Here's a brief rundown of what I did:

1. I cloned the TensorRT repository and navigated to the section for the Stable Diffusion XL demo, as instructed in the README.
2. I followed all the setup and installation instructions in the README to properly prepare my environment and the models for testing. This included setups for both the standard and INT8 quantized inferences.
3. I first ran the model with the standard setup to get a baseline of how fast it performed.
4. Then, I ran the model with INT8 quantization enabled to see how much the performance would improve.

Expected Outcome: Based on NVIDIA's recommendations and claims, I was expecting that turning on INT8 quantization would almost double the performance compared to the standard run.

Actual Outcome: The performance boost from using INT8 quantization was much less than expected. To put it in numbers, without INT8 quantization, the inference took about 2779.89 ms (equating to 0.36 images per second), but with INT8 quantization, it improved slightly to about 2564.51 ms (or 0.39 images per second). This improvement is much smaller than the nearly 2x faster performance I was anticipating, which is a significant difference from what was claimed.

Commands or scripts:
https://github.com/NVIDIA/TensorRT/tree/release/10.0/demo/Diffusion

python3 demo_txt2img_xl.py "a photo of an astronaut riding a horse on mars" --version xl-1.0 --onnx-dir onnx-sdxl --engine-dir engine

python3 demo_txt2img_xl.py "a photo of an astronaut riding a horse on mars" --version xl-1.0 --onnx-dir onnx-sdxl --engine-dir engine-sdxl --int8 --quantization-level 3

Have you tried the latest release?:
Yes, the latest, 10.0.0b6
9.3.0 has the same result.