GTR T5 Large Model output failure in TensorRT 8.6.1 when running on fp16 precision

Description

When I tried to convert gtr t5 model using trtexec command in the TensorRT docker container, the model is getting converted but is giving zero outputs, With fp32 there is no issue.

Environment

TensorRT Version: 8.6.1

NVIDIA GPU: A100

NVIDIA Driver Version: 525.85.12

CUDA Version: 12.1

CUDNN Version:

Operating System:

Python Version (if applicable): 3.10

Tensorflow Version (if applicable):

PyTorch Version (if applicable): 2.0.1

Baremetal or Container (if so, version): 23.07-py3 tensorrt

Steps To Reproduce

1. Convert gtr-t5-large model from Hugging face to onnx using optimum-cli
2. Convert the model from onnx to trt using trtexec with fp16 flag enabled.
3. Infer the text, and we will get zeroes.

As most likely the issue is with overflow LayerNorm, Tried configurations like precisionConstraints (prefer, obey), converting to onnx using latest opset but still the output is coming as 0's and the issue persists.