GPU Latency failure for FP16, INT8, mixed precision (FP16+INT8) models of TensorRT 8.6 when running trtexec on GPU A100

Description

Hello,
I'm trying to do a torch -> onnx -> trt model conversion.
I am doing operations to convert to fp16, to int8 and to mixed precision (fp16 + int8). However, after the conversion is completed, the latency of the fp16 model turns out to be the smallest. Which means fp16 model is faster than int8 and mixed-precision models. Why is that?

Environment

TensorRT Version: 8.6

NVIDIA GPU: A100

NVIDIA Driver Version: 530.30.02

CUDA Version: 12.1

CUDNN Version:

Operating System: Ubuntu 22.04

Python Version (if applicable): 3.10

PyTorch Version (if applicable): 2.1

Baremetal or Container (if so, version): nvcr.io/nvidia/pytorch:23.08-py3

Relevant Files

Model link: "vit_base_patch32_224_clip_laion2b" model from timm.models

Steps To Reproduce

1. Using the pytorch_quantization library we do:

quant_modules.initialize()

quant_desc = QuantDescriptor(num_bits=16)
quant_nn.QuantConv2d.set_default_quant_desc_input(quant_desc)
quant_nn.QuantLinear.set_default_quant_desc_input(quant_desc)
quant_nn.QuantConv2d.set_default_quant_desc_weight(quant_desc)
quant_nn.QuantLinear.set_default_quant_desc_weight(quant_desc)

2. Create a model object in Python (FakeQuant nodes are added automatically because of quant_modules.initialize() line).

m_name = "vit_base_patch32_224_clip_laion2b"
qat_model = create_model(m_name, num_classes=8, exportable=True)

(optionally)
If precision is not fp16, but int8, then specify num_bits=8 in point 1 like that:

quant_desc = QuantDescriptor(num_bits=8)

(optionally)
If the situation is with mixed_precision, then initially we create num_bits=16, then selectively for individual layers we change the values of input_quantizer and weight_quantizer to 8-bit like this:

qat_model.patch_embed.proj._input_quantizer = TensorQuantizer(quant_desc=QuantDescriptor(num_bits=8))

3. We calibrate FakeQuant nodes and do QAT.

4. Do torch.onnx.export.

5. Simplify the onnx model through

onnx_model = onnx.load(os.path.join(SAVE_PATH, "<model_name>.onnx"))
model_simp, check = onnx_simplifier.simplify(onnx_model, check_n=0)
onnx.save(model_simp, os.path.join(SAVE_PATH, "<model_name>.onnx"))

6. Then we convert onnx to trt using the trtexec utility.
   If it is fp16 or int8 precision, then as follows:
   fp16:

trtexec\
     --onnx={os.path.join(SAVE_PATH, '<model_name>.onnx')} \
     --minShapes=input:1x3x224x224 \
     --optShapes=input:10x3x224x224 \
     --maxShapes=input:64x3x224x224 \
     --explicitBatch\
     --saveEngine={os.path.join(SAVE_PATH, '<model_name>.trt')} \
     --exportTimes={os.path.join(SAVE_PATH, 'timing_results.json')} \
     --inputIOFormats=fp16:chw --outputIOFormats=fp16:chw --fp16

int8:

trtexec\
     --onnx={os.path.join(SAVE_PATH, '<model_name>.onnx')} \
     --minShapes=input:1x3x224x224 \
     --optShapes=input:10x3x224x224 \
     --maxShapes=input:64x3x224x224 \
     --explicitBatch\
     --saveEngine={os.path.join(SAVE_PATH, '<model_name>.trt')} \
     --exportTimes={os.path.join(SAVE_PATH, 'timing_results.json')} \
     --inputIOFormats=fp16:chw --outputIOFormats=fp16:chw --int8

If this is mixed-precision, then first we create the str variable "LAYERS_PRECISION" and collect precision for layers in it, iterating over the onnx layers of the model.
The result is something like:
LAYERS_PRECISION="layer1:int8,layer2:int8,layer3:fp16,...,layerN:fp16,"
And then we execute the following command

trtexec\
     --onnx={os.path.join(SAVE_PATH, '<model_name>.onnx')} \
     --fp16 --int8 \
     --precisionConstraints=obey --layerPrecisions={LAYERS_PRECISION} \
     --minShapes=input:1x3x224x224 \
     --optShapes=input:10x3x224x224 \
     --maxShapes=input:64x3x224x224 \
     --explicitBatch\
     --inputIOFormats=fp16:chw --outputIOFormats=fp16:chw \
     --saveEngine={os.path.join(SAVE_PATH, '<model_name>.trt')}

Having done all of the above, we get trt files, which, when checked both through trtexec and through the model-analyzer utility for trt-server, show that the operating speed of the int8 and mixed-precision models is worse than that of the fp16 model.

Commands or scripts: see above

Have you tried the latest release?: yes

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): N/A