Failed to apply int8 quantize Bert model on GPU: "Int8 calibration in TensorRT won't be used on networks containing boolean operators or loops. Please use quantization-aware training to generate a network with Quantize/Dequantize nodes."

Hi,

I adapted the SQUAD example in this repo to use quantization through TRT on a large Roberta model from Hugginface on classification task.
I got everything work (calibration, etc.) without error / warning, until... inference on TRT.
The model works both with onnxruntime CUDA engine and TRT engine (+20% perf compared to optimized ORT) when inferring with FP16 precision.

When I try to enable the int8 quantization, using the model produced by QDQQuantizer I get this message:

Int8 calibration in TensorRT won't be used on networks containing boolean operators or loops. Please use quantization-aware training to generate a network with Quantize/Dequantize nodes.

Even the command trtexec --onnx="./qdq_model.onnx" --int8 --verbose produces the same message.

I don't understand:

• how it's possible, I was thinking that boolean, loops, etc. where replaced by static value during the pytorch -> onnx conversion
• why I can't find this error message in both onnxruntim repo and tensorrt repo

I am using nvcr.io/nvidia/tensorrt:21.09-py3 docker image (TRT 8.03) and master ORT branch.

I have been forced to replace the line

# produce an error message: auto_model.roberta.encoder.layer.23.output.LayerNorm.bias_quantized: invalid weights type of Int8
_ = quantizer.quantize_model()

by

# no error message
quantizer.quantize_tensors()
quantizer.quantize_weights_per_channel()
quantizer.quantize_bias_tensors()
quantizer.remove_nodes()
quantizer.remove_quantized_weights()

To conclude, on Netron, I don't see any for loop / boolean stuff.

Please find attached trtexec logs. I can share the model if needed.

quantization.log

@chilo-ms / @stevenlix : you may want to add in the Bert example a comment saying that the onnx model should be cleaned with symbolic_shape_infer.py script before calibration, to avoid plenty of warnings from MinMax calibration method. Moreover, as said in issue #39 , you may want to pinpoint flatbuffers==1.12 as the V2 has another API (or adapt ORT code)