Diffusion clip and unet refit with fp16 get nan output

Description

I want to test refit using the flowing command. But it is failed.
polygraphy run ./onnx/clip.onnx --input-shapes input_ids:[1,77] --trt --validate --save-engine clip.trt --iterations 1 --fp16 --refittable

This is the output:
[I] RUNNING | Command: /usr/local/bin/polygraphy run ./onnx/clip.onnx --input-shapes input_ids:[1,77] --trt --validate --save-engine clip.trt --iterations 1 --fp16 --refittable
[I] Will generate inference input data according to provided TensorMetadata: {input_ids [shape=(1, 77)]}
[I] trt-runner-N0-04/25/23-09:02:56 | Activating and starting inference
[W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading
[W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[W] onnx2trt_utils.cpp:70: TensorRT is using FLOAT32 precision to run an INT32 ArgMax / ArgMin. Rounding errors may occur for large integer values
[W] Tensor DataType is determined at build time for tensors not marked as input or output.
[I] Configuring with profiles: [Profile().add('input_ids', min=[1, 77], opt=[1, 77], max=[1, 77])]
[I] Building engine with configuration:
Flags | [FP16, REFIT]
Engine Capability | EngineCapability.DEFAULT
Memory Pools | [WORKSPACE: 22592.06 MiB, TACTIC_DRAM: 22592.06 MiB]
Tactic Sources | [CUBLAS, CUBLAS_LT, CUDNN, EDGE_MASK_CONVOLUTIONS, JIT_CONVOLUTIONS]
Profiling Verbosity | ProfilingVerbosity.DETAILED
Preview Features | [FASTER_DYNAMIC_SHAPES_0805, DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805]
[W] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[W] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[W] Check verbose logs for the list of affected weights.
[W] - 110 weights are affected by this issue: Detected subnormal FP16 values.
[W] - 2 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[W] - 1 weights are affected by this issue: Detected finite FP32 values which would overflow in FP16 and converted them to the closest finite FP16 value.
[I] Finished engine building in 29.465 seconds
[I] Saving engine to clip.trt
[I] trt-runner-N0-04/25/23-09:02:56
---- Inference Input(s) ----
{input_ids [dtype=int32, shape=(1, 77)]}
[I] trt-runner-N0-04/25/23-09:02:56
---- Inference Output(s) ----
{text_embeddings [dtype=float32, shape=(1, 77, 768)],
pooler_output [dtype=float32, shape=(1, 768)]}
[I] trt-runner-N0-04/25/23-09:02:56 | Completed 1 iteration(s) in 1.599 ms | Average inference time: 1.599 ms.
[I] Output Validation | Runners: ['trt-runner-N0-04/25/23-09:02:56']
[I] trt-runner-N0-04/25/23-09:02:56 | Validating output: text_embeddings (check_inf=True, check_nan=True)
[I] mean=nan, std-dev=nan, var=nan, median=nan, min=nan at (0, 0, 0), max=nan at (0, 0, 0), avg-magnitude=nan
[E] NaN Detected | One or more NaNs were encountered in this output
[I] Note: Use -vv or set logging verbosity to EXTRA_VERBOSE to display locations of NaNs
[E] Inf Detected | One or more non-finite values were encountered in this output
[I] Note: Use -vv or set logging verbosity to EXTRA_VERBOSE to display non-finite values
[E] FAILED | Errors detected in output: text_embeddings
[I] trt-runner-N0-04/25/23-09:02:56 | Validating output: pooler_output (check_inf=True, check_nan=True)
[I] mean=nan, std-dev=nan, var=nan, median=nan, min=nan at (0, 0), max=nan at (0, 0), avg-magnitude=nan
[E] NaN Detected | One or more NaNs were encountered in this output
[E] Inf Detected | One or more non-finite values were encountered in this output
[E] FAILED | Errors detected in output: pooler_output
[E] FAILED | Output Validation
[E] FAILED | Runtime: 38.321s | Command: /usr/local/bin/polygraphy run ./onnx/clip.onnx --input-shapes input_ids:[1,77] --trt --validate --save-engine clip.trt --iterations 1 --fp16 --refittable

Environment

TensorRT Version: 8.6
NVIDIA GPU: A10
NVIDIA Driver Version: 515.86.01
CUDA Version: 12.1
CUDNN Version:
Operating System: ubuntu 20.04
Python Version (if applicable): 3.10
Tensorflow Version (if applicable):
PyTorch Version (if applicable): 1.13
Baremetal or Container (if so, version):

Relevant Files

Steps To Reproduce