fp16 conversion breaks model

Description

fp16 conversion using polygraphy breaks model. for onnx fp16 conversion, the inference results is not exactly the same but close enough. it works as expected

Environment

TensorRT Version: 8.5.2.2

NVIDIA GPU: xavier nx

CUDA Version: 12.2

Operating System: ubuntu 20.04

Python Version (if applicable): 3.12.4

Relevant Files

Model link:
https://drive.google.com/file/d/1HkNSzST8aGP9ZYyDMg8o-l3tTJhVRf68/view?usp=sharing

Steps To Reproduce

import numpy as np
from polygraphy.backend.trt import (
    CreateConfig,
    EngineFromNetwork,
    NetworkFromOnnxPath,
    SaveEngine,
    TrtRunner,
)
import onnx
from onnxruntime import InferenceSession
import onnxruntime
from onnxconverter_common import float16

onnx_model = "parseq_recognizer_fix.onnx"

def main():
    model = onnx.load(onnx_model)
    model_fp16 = float16.convert_float_to_float16(model, keep_io_types=True)
    onnx.save(model_fp16, "fp16_"+onnx_model)

    inp_data = np.ones(shape=(1, 3, 32, 128), dtype=np.float32)
    rsess = InferenceSession(onnx_model, 
        providers=["CUDAExecutionProvider"]) 
    pred = rsess.run(None, {"input": inp_data})
    print(pred)
    rsess = InferenceSession("fp16_"+onnx_model, 
        providers=["CUDAExecutionProvider"]) 
    pred = rsess.run(None, {"input": inp_data})
    print(pred)

    build_engine = EngineFromNetwork(
        NetworkFromOnnxPath(onnx_model), config=CreateConfig(fp16=False)
    )  # Note that config is an optional argument.
    build_engine = SaveEngine(build_engine, path="parseq_test.engine")
    with TrtRunner(build_engine) as runner:
        outputs = runner.infer(feed_dict={"input": inp_data})
        print(outputs)

    build_engine = EngineFromNetwork(
        NetworkFromOnnxPath(onnx_model), config=CreateConfig(fp16=True)
    )  # Note that config is an optional argument.
    build_engine = SaveEngine(build_engine, path="parseq_test_fp16.engine")
    with TrtRunner(build_engine) as runner:
        outputs = runner.infer(feed_dict={"input": inp_data})
        print(outputs)


if __name__ == "__main__":
    main()

output:

2024-10-01 05:25:51.603274671 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 194556, index: 0, mask: {5, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-10-01 05:25:54.690404095 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 54 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2024-10-01 05:25:54.735311715 [W:onnxruntime:, session_state.cc:1166 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2024-10-01 05:25:54.735495111 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2024-10-01 05:25:57.397336427 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 194568, index: 0, mask: {5, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-10-01 05:26:00.073771102 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 54 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2024-10-01 05:26:00.114621624 [W:onnxruntime:, session_state.cc:1166 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2024-10-01 05:26:00.114811420 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
/home/dc/.local/lib/python3.8/site-packages/onnxconverter_common/float16.py:43: UserWarning: the float32 number 9.080395102500916e-08 will be truncated to 1e-07
  warnings.warn("the float32 number {} will be truncated to {}".format(pos_min, min_positive_val))
/home/dc/.local/lib/python3.8/site-packages/onnxconverter_common/float16.py:53: UserWarning: the float32 number -3.236345946788788e-08 will be truncated to -1e-07
  warnings.warn("the float32 number {} will be truncated to {}".format(neg_max, -min_positive_val))
/home/dc/.local/lib/python3.8/site-packages/onnxconverter_common/float16.py:43: UserWarning: the float32 number 6.658956408500671e-08 will be truncated to 1e-07
  warnings.warn("the float32 number {} will be truncated to {}".format(pos_min, min_positive_val))
/home/dc/.local/lib/python3.8/site-packages/onnxconverter_common/float16.py:53: UserWarning: the float32 number -4.493631422519684e-08 will be truncated to -1e-07
  warnings.warn("the float32 number {} will be truncated to {}".format(neg_max, -min_positive_val))
/home/dc/.local/lib/python3.8/site-packages/onnxconverter_common/float16.py:43: UserWarning: the float32 number 4.237517714500427e-08 will be truncated to 1e-07
  warnings.warn("the float32 number {} will be truncated to {}".format(pos_min, min_positive_val))
/home/dc/.local/lib/python3.8/site-packages/onnxconverter_common/float16.py:43: UserWarning: the float32 number 9.42964106798172e-08 will be truncated to 1e-07
  warnings.warn("the float32 number {} will be truncated to {}".format(pos_min, min_positive_val))
/home/dc/.local/lib/python3.8/site-packages/onnxconverter_common/float16.py:50: UserWarning: the float32 number -inf will be truncated to -10000.0
  warnings.warn("the float32 number {} will be truncated to {}".format(neg_min, -max_finite_val))
[array([[0.31324247, 0.19727543, 0.3371198 , 0.25981265, 0.32797444,
        0.28933585, 0.3222586 , 0.23759457, 0.23015757, 0.22676414,
        0.26784003, 0.39013135, 0.2612777 , 0.27258867, 0.24250458,
        0.29773933, 0.31730056, 0.3611088 , 0.3215531 , 0.24237499,
        0.16885404, 0.2338847 , 0.24125989, 0.25620067, 0.20599137,
        0.57806915]], dtype=float32), array([[2, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 0]], dtype=int32)]
[array([[0.27319336, 0.4206543 , 0.2980957 , 0.265625  , 0.35009766,
        0.2993164 , 0.32543945, 0.23986816, 0.23327637, 0.22192383,
        0.26367188, 0.36499023, 0.27172852, 0.28344727, 0.24584961,
        0.30151367, 0.29296875, 0.35327148, 0.31567383, 0.24414062,
        0.17712402, 0.2541504 , 0.23999023, 0.27856445, 0.22766113,
        0.31689453]], dtype=float32), array([[2, 1, 1, 0, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1,
        1, 1, 1, 0]], dtype=int32)]
[W] onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[W] onnx2trt_utils.cpp:403: One or more weights outside the range of INT32 was clamped
[W] Tensor DataType is determined at build time for tensors not marked as input or output.
[I] Configuring with profiles:[
        Profile 0:
            {input [min=[1, 3, 32, 128], opt=[1, 3, 32, 128], max=[1, 3, 32, 128]]}
    ]
[I] Building engine with configuration:
    Flags                  | []
    Engine Capability      | EngineCapability.DEFAULT
    Memory Pools           | [WORKSPACE: 6854.13 MiB]
    Tactic Sources         | [CUBLAS, CUBLAS_LT, CUDNN, EDGE_MASK_CONVOLUTIONS, JIT_CONVOLUTIONS]
    Profiling Verbosity    | ProfilingVerbosity.DETAILED
[I] Finished engine building in 97.692 seconds
[I] Saving engine to parseq_test.engine
OrderedDict([('out', array([[0.31324273, 0.1972756 , 0.3371203 , 0.25981286, 0.32797468,
        0.28933626, 0.32225883, 0.23759452, 0.23015758, 0.22676419,
        0.2678403 , 0.39013153, 0.2612777 , 0.27258864, 0.2425049 ,
        0.29773965, 0.31730086, 0.36110893, 0.32155344, 0.24237521,
        0.16885436, 0.23388492, 0.24126002, 0.25620082, 0.20599148,
        0.5780687 ]], dtype=float32)), ('12658', array([[2, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 0]], dtype=int32))])
[I] Configuring with profiles:[
        Profile 0:
            {input [min=[1, 3, 32, 128], opt=[1, 3, 32, 128], max=[1, 3, 32, 128]]}
    ]
[I] Building engine with configuration:
    Flags                  | [FP16]
    Engine Capability      | EngineCapability.DEFAULT
    Memory Pools           | [WORKSPACE: 6854.13 MiB]
    Tactic Sources         | [CUBLAS, CUBLAS_LT, CUDNN, EDGE_MASK_CONVOLUTIONS, JIT_CONVOLUTIONS]
    Profiling Verbosity    | ProfilingVerbosity.DETAILED
[W] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[W] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[W] Check verbose logs for the list of affected weights.
[W] - 2 weights are affected by this issue: Detected FP32 infinity values and converted them to corresponding FP16 infinity.
[W] - 82 weights are affected by this issue: Detected subnormal FP16 values.
[W] - 3 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[I] Finished engine building in 230.183 seconds
[I] Saving engine to parseq_test_fp16.engine
OrderedDict([('out', array([[1.        , 1.        , 1.        , 1.        , 1.        ,
        0.39941406, 0.15283203, 0.2734375 , 0.35595703, 0.27124023,
        0.16320801, 0.21789551, 0.14758301, 0.23620605, 0.2565918 ,
        0.17480469, 0.4111328 , 0.41455078, 0.4189453 , 0.5644531 ,
        0.75683594, 0.32006836, 0.59033203, 0.42797852, 0.37353516,
        0.80322266]], dtype=float32)), ('12658', array([[50, 25, 24, 15,  0, 25, 29, 15, 15, 15, 15, 15, 29, 15, 15, 15,
        15, 15, 15, 15, 15, 15, 15, 24, 15, 24]], dtype=int32))])

FURTHERMORE multiple inferences on parseq_test_fp16.engine using different random input also consistenly produces the same output

Begin with the supplied Python reproduction and linked model; compare the ONNX Runtime FP16 result with the TensorRT FP16 engine output, focusing on the warnings about infinity and subnormal weights. Done means identifying the responsible conversion stage and demonstrating consistent output with a focused regression case.