“Error[2]: [optimizer.cpp::match::959] Error Code 2: Internal Error (Assertion !n23->cost.empty() failed. Node: )” of TensorRT 8.4 when running QDQ onnx on orin

Description

I tried to transfer onnx file to tensorRT for deployment, but it fails with the error below .

08/01/2024-21:54:09] [V] [TRT] *************** Autotuning format combination: Half(256,256,1), Half(1), Half(1), Half() -> Half(256:2,256,1) *************** [08/01/2024-21:54:09] [V] [TRT] --------------- Timing Runner: NvLayerNormPlugin_6 (PluginV2) [08/01/2024-21:54:09] [V] [TRT] Tactic: 0 Time: 0.00653777 [08/01/2024-21:54:09] [V] [TRT] Fastest Tactic: 0 Time: 0.00653777 [08/01/2024-21:54:09] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PluginV2 Tactic: 0 [08/01/2024-21:54:09] [V] [TRT] *************** Autotuning format combination: Half(256,256,1), Half(1), Half(1), Half() -> Half(1:4,27392,107) *************** [08/01/2024-21:54:09] [V] [TRT] --------------- Timing Runner: NvLayerNormPlugin_6 (PluginV2) [08/01/2024-21:54:09] [V] [TRT] Tactic: 0 Time: 0.00639503 [08/01/2024-21:54:09] [V] [TRT] Fastest Tactic: 0 Time: 0.00639503 [08/01/2024-21:54:09] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PluginV2 Tactic: 0 [08/01/2024-21:54:09] [V] [TRT] *************** Autotuning format combination: Half(256,256,1), Half(1), Half(1), Half() -> Half(1:8,13824,54) *************** [08/01/2024-21:54:09] [V] [TRT] --------------- Timing Runner: NvLayerNormPlugin_6 (PluginV2) [08/01/2024-21:54:09] [V] [TRT] Tactic: 0 Time: 0.00652426 [08/01/2024-21:54:09] [V] [TRT] Fastest Tactic: 0 Time: 0.00652426 [08/01/2024-21:54:09] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PluginV2 Tactic: 0 [08/01/2024-21:54:09] [V] [TRT] *************** Autotuning format combination: Half(256,256,1), Half(1), Half(1), Half() -> Half(1:16,6912,27) *************** [08/01/2024-21:54:09] [V] [TRT] --------------- Timing Runner: NvLayerNormPlugin_6 (PluginV2) [08/01/2024-21:54:09] [V] [TRT] Tactic: 0 Time: 0.00647752 [08/01/2024-21:54:09] [V] [TRT] Fastest Tactic: 0 Time: 0.00647752 [08/01/2024-21:54:09] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PluginV2 Tactic: 0 [08/01/2024-21:54:09] [V] [TRT] *************** Autotuning format combination: Half(256,256,1), Half(1), Half(1), Half() -> Half(256:32,256,1) *************** [08/01/2024-21:54:09] [V] [TRT] --------------- Timing Runner: NvLayerNormPlugin_6 (PluginV2) [08/01/2024-21:54:09] [V] [TRT] Tactic: 0 Time: 0.00645585 [08/01/2024-21:54:09] [V] [TRT] Fastest Tactic: 0 Time: 0.00645585 [08/01/2024-21:54:09] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: PluginV2 Tactic: 0 [08/01/2024-21:54:09] [V] [TRT] =============== Computing costs for [08/01/2024-21:54:09] [V] [TRT] *************** Autotuning format combination: Float(1), Float(16,16,4,1), Float(10240,16,4,1), Float(1), Float(256,256,1), Float(256,256,1), Float(256,256,1), Float(256,256,1), Float(256,256,1), Float(256,256,1), Float(1284,3,1) -> Float(8192,16,4,1), Float(109568,256,1), Float(428,428,1,1), Float(4708,4708,11,1), Float(7276,7276,17,1) *************** [08/01/2024-21:54:09] [V] [TRT] --------------- Timing Runner: {ForeignNode[Reshape_247...Concat_2246]} (Myelin) [08/01/2024-21:54:14] [V] [TRT] Tactic: 0 Time: 1.47548 [08/01/2024-21:54:14] [V] [TRT] Fastest Tactic: 0 Time: 1.47548 [08/01/2024-21:54:14] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Myelin Tactic: 0 [08/01/2024-21:54:14] [V] [TRT] *************** Autotuning format combination: Half(1), Half(16,16,4,1), Half(10240,16,4,1), Half(1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(1284,3,1) -> Half(8192,16,4,1), Half(109568,256,1), Half(428,428,1,1), Half(4708,4708,11,1), Half(7276,7276,17,1) *************** [08/01/2024-21:54:14] [V] [TRT] --------------- Timing Runner: {ForeignNode[Reshape_247...Concat_2246]} (Myelin) [08/01/2024-21:54:19] [V] [TRT] Tactic: 0 Time: 1.52296 [08/01/2024-21:54:19] [V] [TRT] Fastest Tactic: 0 Time: 1.52296 [08/01/2024-21:54:19] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Myelin Tactic: 0 [08/01/2024-21:54:19] [V] [TRT] *************** Autotuning format combination: Half(1), Half(16,1:8,4,1), Half(1280,1:8,320,80), Half(1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(256,256,1), Half(1284,3,1) -> Half(1024,1:8,256,64), Half(109568,256,1), Half(428,1:8,1,1), Half(4708,1:8,11,1), Half(7276,1:8,17,1) *************** [08/01/2024-21:54:19] [V] [TRT] --------------- Timing Runner: {ForeignNode[Reshape_247...Concat_2246]} (Myelin) [08/01/2024-21:54:24] [V] [TRT] Tactic: 0 Time: 1.58639 [08/01/2024-21:54:24] [V] [TRT] Fastest Tactic: 0 Time: 1.58639 [08/01/2024-21:54:24] [V] [TRT] >>>>>>>>>>>>>>> Chose Runner Type: Myelin Tactic: 0 [08/01/2024-21:54:24] [V] [TRT] Deleting timing cache: 4667 entries, served 34843 hits since creation. [08/01/2024-21:54:24] [E] Error[2]: [optimizer.cpp::match::959] Error Code 2: Internal Error (Assertion !n23->cost.empty() failed. Node: ) [08/01/2024-21:54:24] [E] Error[2]: [builder.cpp::buildSerializedNetwork::619] Error Code 2: Internal Error (Assertion engine != nullptr failed. ) [08/01/2024-21:54:24] [E] Engine could not be created from network [08/01/2024-21:54:24] [E] Building engine failed [08/01/2024-21:54:24] [E] Failed to create engine from model or file. [08/01/2024-21:54:24] [E] Engine set up failed

Environment

TensorRT Version: 8.4

NVIDIA GPU: orin

CUDA Version: 11.4.r11.4

Steps To Reproduce

Commands or scripts: ./trtexec --onnx= sim_online_calib256_int8_percent_matmul_pertensor_with_blacklist__stage2_qdq_rep_ln_rep_qkv.onnx --dumpProfile=true --verbose=true --int8 --fp16

Start with the supplied ./trtexec command, the QDQ ONNX model named sim_online_calib256_int8_percent_matmul_pertensor_with_blacklist__stage2_qdq_rep_qkv.onnx, and the verbose optimizer log. Reproduce on TensorRT 8.4 with the stated Orin and CUDA 11.4 environment, then narrow the failure around optimizer.cpp::match and the QDQ network. Done means the engine builds successfully or the failing model pattern and reproducible cause are identified.