Marginal Improvement Between INT8 and FP16

I have INT8 quantized a BERT model for binary text classification and am only getting a marginal improvement in speed over FP16.

I am using the transformer-deploy library that utilizes TensorRT.

Tested on both an A4000 and A100 GPU.

A4000 --> TensorRT INT-8: 34.48ms, TensorRT FP16: 38.72ms
A100 ---> TensorRT INT-8: 11.53ms, TensorRT FP16: 11.75ms

These are the components that were quant disabled to improve accuracy:

disable bert.encoder.layer.1.intermediate.dense._input_quantizer
disable bert.encoder.layer.2.attention.output.layernorm_quantizer_0
disable bert.encoder.layer.2.attention.output.layernorm_quantizer_1
disable bert.encoder.layer.2.output.layernorm_quantizer_0
disable bert.encoder.layer.2.output.layernorm_quantizer_1
disable bert.encoder.layer.3.attention.output.dense._input_quantizer
disable bert.encoder.layer.10.attention.self.key._input_quantizer
disable bert.encoder.layer.11.attention.output.dense._input_quantizer
disable bert.encoder.layer.11.output.dense._input_quantizer

The debug logs from the A4000 run are attached here:

trt_logs_int8_quantization.txt

I also tried using the Profiler during the inference of one sample in case that provided any useful information but it seemed to only print out one layer. I assume it should print out info on many layers?

profiler = trt.Profiler()
context.profiler = profiler   

context: IExecutionContext = engine.create_execution_context()
context.set_optimization_profile_async(
    profile_index=profile_index, stream_handle=torch.cuda.current_stream().cuda_stream
)
input_binding_idxs, output_binding_idxs = get_binding_idxs(engine, profile_index)  # type: List[int], List[int]

data = train_tokenized[0:1]
input_torch: OD[str, torch.Tensor] = convert_tensor(data=data, output="torch")
input_np: OD[str, np.ndarray] = convert_tensor(data=data, output="np")

tensorrt_output = infer_tensorrt(
    context=context,
    inputs=input_torch,
    input_binding_idxs=input_binding_idxs,
    output_binding_idxs=output_binding_idxs,
)

-----------------------------------------------------------------------------------------

[HostToDeviceCopy]: 0.018464ms
{ForeignNode[bert.embeddings.position_embeddings.weight...(Unnamed Layer* 2639) [ElementWise]]}: 1.73869ms

Any insight into these results is greatly appreciated. Thank you.

Versions:
Python: 3.10.9
transformers-deploy: 0.5.4
TensorRT: 8.4.1.5
Onnxruntime (GPU): 1.12.0
Cuda: 11.7