TensorRT provider does not use the cache it created

### Describe the issue

The symptoms I see is that if I delete the cache it takes 9 seconds to regenerate the cache files. If I create more engines for the same model in the same process it takes 40 ms or so. But if I restart the program it takes 9 seconds again _but the cache files are not rewritten_. This does not seem normal.

I turned on verbose logging and it does not show any errors. Attached the log below.

The only suspicious output is: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. I didn't feel this was important as a) we definitely only have F32 weights, b) calculations are carried out correctly subsequently. Maybe also: [TensorRT EP] Model path is empty which could possible be related to us providing the ONNX data as memory buffer to the Ort::Session constructor. I have no idea how the "hash" part of the .engine and .profile files is constructed but I would guess it comes from the modelData.

The ONNX file I use has a network that accepts different input sizes (and generates corresponding output sizes). I only use it with one size in our current application. Does this still mean that without specifying trt_profile_min_shapes etc. the cache is not used? Even if it wasn't I don't think it is consistent with the symptoms.

Here is your log output:
core 1 consist of logical processors: 1 2
core 2 consist of logical processors: 3 4
core 3 consist of logical processors: 5 6
core 4 consist of logical processors: 7 8
core 5 consist of logical processors: 9 10
core 6 consist of logical processors: 11 12
core 7 consist of logical processors: 13 14
core 8 consist of logical processors: 15 16
core 9 consist of logical processors: 17
core 10 consist of logical processors: 18
core 11 consist of logical processors: 19
core 12 consist of logical processors: 20
core 13 consist of logical processors: 21
core 14 consist of logical processors: 22
core 15 consist of logical processors: 23
core 16 consist of logical processors: 24
[TensorRT EP] Getting all registered TRT plugins from TRT plugin registry ...
[TensorRT EP] CaskDeconvShaderWeightsTransformerPlugin, version : 1
[TensorRT EP] CaskConvShaderWeightsTransformerPlugin, version : 1
[TensorRT EP] CustomQKVToContextPluginDynamic, version : 3
[TensorRT EP] CustomQKVToContextPluginDynamic, version : 1
[TensorRT EP] CustomQKVToContextPluginDynamic, version : 2
[TensorRT EP] DisentangledAttention_TRT, version : 1
[TensorRT EP] CustomEmbLayerNormPluginDynamic, version : 1
[TensorRT EP] CustomEmbLayerNormPluginDynamic, version : 2
[TensorRT EP] CustomEmbLayerNormPluginDynamic, version : 3
[TensorRT EP] CustomFCPluginDynamic, version : 1
[TensorRT EP] CustomGeluPluginDynamic, version : 1
[TensorRT EP] GroupNormalizationPlugin, version : 1
[TensorRT EP] RnRes2Br1Br2c_TRT, version : 1
[TensorRT EP] RnRes2Br1Br2c_TRT, version : 2
[TensorRT EP] RnRes2Br2bBr2c_TRT, version : 1
[TensorRT EP] RnRes2Br2bBr2c_TRT, version : 2
[TensorRT EP] SingleStepLSTMPlugin, version : 1
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 3
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 4
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 1
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 2
[TensorRT EP] RnRes2FullFusion_TRT, version : 1
[TensorRT EP] BatchedNMSDynamic_TRT, version : 1
[TensorRT EP] BatchedNMS_TRT, version : 1
[TensorRT EP] BatchTilePlugin_TRT, version : 1
[TensorRT EP] Clip_TRT, version : 1
[TensorRT EP] CoordConvAC, version : 1
[TensorRT EP] CropAndResizeDynamic, version : 1
[TensorRT EP] CropAndResize, version : 1
[TensorRT EP] DecodeBbox3DPlugin, version : 1
[TensorRT EP] DetectionLayer_TRT, version : 1
[TensorRT EP] EfficientNMS_Explicit_TF_TRT, version : 1
[TensorRT EP] EfficientNMS_Implicit_TF_TRT, version : 1
[TensorRT EP] EfficientNMS_ONNX_TRT, version : 1
[TensorRT EP] EfficientNMS_TRT, version : 1
[TensorRT EP] FlattenConcat_TRT, version : 1
[TensorRT EP] GenerateDetection_TRT, version : 1
[TensorRT EP] GridAnchor_TRT, version : 1
[TensorRT EP] GridAnchorRect_TRT, version : 1
[TensorRT EP] InstanceNormalization_TRT, version : 1
[TensorRT EP] InstanceNormalization_TRT, version : 2
[TensorRT EP] LReLU_TRT, version : 1
[TensorRT EP] ModulatedDeformConv2d, version : 1
[TensorRT EP] MultilevelCropAndResize_TRT, version : 1
[TensorRT EP] MultilevelProposeROI_TRT, version : 1
[TensorRT EP] MultiscaleDeformableAttnPlugin_TRT, version : 1
[TensorRT EP] NMSDynamic_TRT, version : 1
[TensorRT EP] NMS_TRT, version : 1
[TensorRT EP] Normalize_TRT, version : 1
[TensorRT EP] PillarScatterPlugin, version : 1
[TensorRT EP] PriorBox_TRT, version : 1
[TensorRT EP] ProposalDynamic, version : 1
[TensorRT EP] ProposalLayer_TRT, version : 1
[TensorRT EP] Proposal, version : 1
[TensorRT EP] PyramidROIAlign_TRT, version : 1
[TensorRT EP] Region_TRT, version : 1
[TensorRT EP] Reorg_TRT, version : 1
[TensorRT EP] ResizeNearest_TRT, version : 1
[TensorRT EP] ROIAlign_TRT, version : 1
[TensorRT EP] RPROI_TRT, version : 1
[TensorRT EP] ScatterND, version : 1
[TensorRT EP] SpecialSlice_TRT, version : 1
[TensorRT EP] Split, version : 1
[TensorRT EP] VoxelGeneratorPlugin, version : 1
[TensorRT EP] Getting all registered TRT plugins from TRT plugin registry ...
[TensorRT EP] CaskDeconvShaderWeightsTransformerPlugin, version : 1
[TensorRT EP] CaskConvShaderWeightsTransformerPlugin, version : 1
[TensorRT EP] CustomQKVToContextPluginDynamic, version : 3
[TensorRT EP] CustomQKVToContextPluginDynamic, version : 1
[TensorRT EP] CustomQKVToContextPluginDynamic, version : 2
[TensorRT EP] DisentangledAttention_TRT, version : 1
[TensorRT EP] CustomEmbLayerNormPluginDynamic, version : 1
[TensorRT EP] CustomEmbLayerNormPluginDynamic, version : 2
[TensorRT EP] CustomEmbLayerNormPluginDynamic, version : 3
[TensorRT EP] CustomFCPluginDynamic, version : 1
[TensorRT EP] CustomGeluPluginDynamic, version : 1
[TensorRT EP] GroupNormalizationPlugin, version : 1
[TensorRT EP] RnRes2Br1Br2c_TRT, version : 1
[TensorRT EP] RnRes2Br1Br2c_TRT, version : 2
[TensorRT EP] RnRes2Br2bBr2c_TRT, version : 1
[TensorRT EP] RnRes2Br2bBr2c_TRT, version : 2
[TensorRT EP] SingleStepLSTMPlugin, version : 1
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 3
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 4
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 1
[TensorRT EP] CustomSkipLayerNormPluginDynamic, version : 2
[TensorRT EP] RnRes2FullFusion_TRT, version : 1
[TensorRT EP] BatchedNMSDynamic_TRT, version : 1
[TensorRT EP] BatchedNMS_TRT, version : 1
[TensorRT EP] BatchTilePlugin_TRT, version : 1
[TensorRT EP] Clip_TRT, version : 1
[TensorRT EP] CoordConvAC, version : 1
[TensorRT EP] CropAndResizeDynamic, version : 1
[TensorRT EP] CropAndResize, version : 1
[TensorRT EP] DecodeBbox3DPlugin, version : 1
[TensorRT EP] DetectionLayer_TRT, version : 1
[TensorRT EP] EfficientNMS_Explicit_TF_TRT, version : 1
[TensorRT EP] EfficientNMS_Implicit_TF_TRT, version : 1
[TensorRT EP] EfficientNMS_ONNX_TRT, version : 1
[TensorRT EP] EfficientNMS_TRT, version : 1
[TensorRT EP] FlattenConcat_TRT, version : 1
[TensorRT EP] GenerateDetection_TRT, version : 1
[TensorRT EP] GridAnchor_TRT, version : 1
[TensorRT EP] GridAnchorRect_TRT, version : 1
[TensorRT EP] InstanceNormalization_TRT, version : 1
[TensorRT EP] InstanceNormalization_TRT, version : 2
[TensorRT EP] LReLU_TRT, version : 1
[TensorRT EP] ModulatedDeformConv2d, version : 1
[TensorRT EP] MultilevelCropAndResize_TRT, version : 1
[TensorRT EP] MultilevelProposeROI_TRT, version : 1
[TensorRT EP] MultiscaleDeformableAttnPlugin_TRT, version : 1
[TensorRT EP] NMSDynamic_TRT, version : 1
[TensorRT EP] NMS_TRT, version : 1
[TensorRT EP] Normalize_TRT, version : 1
[TensorRT EP] PillarScatterPlugin, version : 1
[TensorRT EP] PriorBox_TRT, version : 1
[TensorRT EP] ProposalDynamic, version : 1
[TensorRT EP] ProposalLayer_TRT, version : 1
[TensorRT EP] Proposal, version : 1
[TensorRT EP] PyramidROIAlign_TRT, version : 1
[TensorRT EP] Region_TRT, version : 1
[TensorRT EP] Reorg_TRT, version : 1
[TensorRT EP] ResizeNearest_TRT, version : 1
[TensorRT EP] ROIAlign_TRT, version : 1
[TensorRT EP] RPROI_TRT, version : 1
[TensorRT EP] ScatterND, version : 1
[TensorRT EP] SpecialSlice_TRT, version : 1
[TensorRT EP] Split, version : 1
[TensorRT EP] VoxelGeneratorPlugin, version : 1
Flush-to-zero and denormal-as-zero are off
Creating and using per session threadpools since use_per_session_threads_ is true
Dynamic block base set to 0
SetThreadAffinityMask done for thread: 55872, group_id: 0, mask: 12
SetThreadAffinityMask done for thread: 61448, group_id: 0, mask: 48
SetThreadAffinityMask done for thread: 85932, group_id: 0, mask: 192
SetThreadAffinityMask done for thread: 97056, group_id: 0, mask: 768
SetThreadAffinityMask done for thread: 90348, group_id: 0, mask: 12288
SetThreadAffinityMask done for thread: 91864, group_id: 0, mask: 65536
SetThreadAffinityMask done for thread: 50880, group_id: 0, mask: 3072
SetThreadAffinityMask done for thread: 77020, group_id: 0, mask: 262144SetThreadAffinityMask done for thread: 93328, group_id: 0, mask: 2097152
SetThreadAffinityMask done for thread: 70988, group_id: 0, mask: 131072
SetThreadAffinityMask done for thread: 45040, group_id: 0, mask: 49152SetThreadAffinityMask done for thread: 94268, group_id: 0, mask: 524288
SetThreadAffinityMask done for thread: 68884, group_id: 0, mask: 8388608

SetThreadAffinityMask done for thread: 97236, group_id: 0, mask: 4194304
SetThreadAffinityMask done for thread: 99776, group_id: 0, mask: 1048576

# I manually split this into multiple lines, the output was one long line:
[TensorRT EP] TensorRT provider options: device_id: 0,
trt_max_partition_iterations: 1000,
trt_min_subgraph_size: 1,
trt_max_workspace_size: 1073741824,
trt_fp16_enable: 0,
trt_int8_enable: 0,
trt_int8_calibration_cache_name: ,
int8_calibration_cache_available: 0,
trt_int8_use_native_tensorrt_calibration_table: 0,
trt_dla_enable: 0,
trt_dla_core: 0,
trt_dump_subgraphs: 0,
trt_engine_cache_enable: 1,
trt_cache_path: C:\ProgramData\ContextVision\cvn_cache\2beba869a9eb439b3c12a6bb70db02bc7d92381d,
trt_engine_decryption_enable: 0,
trt_engine_decryption_lib_path: ,
trt_force_sequential_engine_build: 0,
trt_context_memory_sharing_enable: 0,
trt_layer_norm_fp32_fallback: 0,
trt_build_heuristics_enable: 0,
trt_sparsity_enable: 0,
trt_builder_optimization_level: 3,
trt_auxiliary_streams: -1,
trt_tactic_sources: ,
trt_profile_min_shapes: ,
trt_profile_max_shapes: ,
trt_profile_opt_shapes: ,
trt_cuda_graph_enable: 0
Initializing session.
Adding default CPU execution provider.
Creating BFCArena for Cuda with following configs: initial_chunk_size_bytes: 1048576 max_dead_bytes_per_chunk: 134217728 initial_growth_chunk_size_bytes: 2097152 max_power_of_two_extend_bytes: 1073741824 memory limit: 18446744073709551615 arena_extend_strategy: 0
Creating 21 bins of max chunk size 256 to 268435456
Creating BFCArena for CudaPinned with following configs: initial_chunk_size_bytes: 1048576 max_dead_bytes_per_chunk: 134217728 initial_growth_chunk_size_bytes: 2097152 max_power_of_two_extend_bytes: 1073741824 memory limit: 18446744073709551615 arena_extend_strategy: 0
Creating 21 bins of max chunk size 256 to 268435456
Creating BFCArena for Cpu with following configs: initial_chunk_size_bytes: 1048576 max_dead_bytes_per_chunk: 134217728 initial_growth_chunk_size_bytes: 2097152 max_power_of_two_extend_bytes: 1073741824 memory limit: 18446744073709551615 arena_extend_strategy: 0
Creating 21 bins of max chunk size 256 to 268435456
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer TransposeOptimizer modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
GraphTransformer Level1_RuleBasedTransformer modified: 0 with status: OK
GraphTransformer DoubleQDQPairsRemover modified: 0 with status: OK
GraphTransformer ConstantSharing modified: 1 with status: OK
GraphTransformer CommonSubexpressionElimination modified: 0 with status: OK
GraphTransformer ConstantFolding modified: 0 with status: OK
GraphTransformer MatMulAddFusion modified: 0 with status: OK
GraphTransformer ReshapeFusion modified: 0 with status: OK
GraphTransformer FreeDimensionOverrideTransformer modified: 0 with status: OK
GraphTransformer QDQPropagationTransformer modified: 0 with status: OK
GraphTransformer EnsureUniqueDQForNodeUnit modified: 0 with status: OK
GraphTransformer RocmBlasAltImpl modified: 0 with status: OK
[TensorRT EP] Model path is empty
[2023-11-03 12:17:33 WARNING] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT EP] TensorRT subgraph MetaDef name TRTKernel_graph_torch-jit-export_6809225732474184598_0
[TensorRT EP] TensorRT subgraph MetaDef name TRTKernel_graph_torch-jit-export_6809225732474184598_0
[TensorRT EP] Whole graph will run on TensorRT execution provider
[2023-11-03 12:17:36 WARNING] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
Removing initializer 'blocks.15.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.0.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.15.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.0.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.2.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.18.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.2.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.11.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.4.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.11.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.4.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.13.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.6.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.29.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.13.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.6.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.35.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.9.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.35.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.9.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.18.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.20.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.20.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.22.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.22.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.24.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.41.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.24.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.27.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.27.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.29.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.31.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.31.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.33.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.33.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.38.conv.weight'. It is no longer used by any node.
Removing initializer 'blocks.38.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.41.conv.bias'. It is no longer used by any node.
Removing initializer 'blocks.44.conv.weight'. It is no longer used by any node.
Removing initializer 'ortshared_1_1_4_0_token_8'. It is no longer used by any node.
GraphTransformer TransposeOptimizer_CPUExecutionProvider modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer QDQS8ToU8Transformer modified: 0 with status: OK
GraphTransformer QDQSelectorActionTransformer modified: 0 with status: OK
GraphTransformer GemmActivationFusion modified: 0 with status: OK
GraphTransformer MatMulIntegerToFloatFusion modified: 0 with status: OK
GraphTransformer DynamicQuantizeMatMulFusion modified: 0 with status: OK
GraphTransformer ConvActivationFusion modified: 0 with status: OK
GraphTransformer GeluFusion modified: 0 with status: OK
GraphTransformer LayerNormFusion modified: 0 with status: OK
GraphTransformer SimplifiedLayerNormFusion modified: 0 with status: OK
GraphTransformer AttentionFusion modified: 0 with status: OK
GraphTransformer EmbedLayerNormFusion modified: 0 with status: OK
GraphTransformer GatherToSplitFusion modified: 0 with status: OK
GraphTransformer GatherToSliceFusion modified: 0 with status: OK
GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
GraphTransformer BiasGeluFusion modified: 0 with status: OK
GraphTransformer SkipLayerNormFusion modified: 1 with status: OK
GraphTransformer FastGeluFusion modified: 0 with status: OK
GraphTransformer QuickGeluFusion modified: 0 with status: OK
GraphTransformer BiasSoftmaxFusion modified: 0 with status: OK
GraphTransformer BiasDropoutFusion modified: 0 with status: OK
GraphTransformer MatMulScaleFusion modified: 0 with status: OK
GraphTransformer MatMulActivationFusion modified: 0 with status: OK
GraphTransformer QDQFinalCleanupTransformer modified: 0 with status: OK
GraphTransformer NchwcTransformer modified: 0 with status: OK
GraphTransformer NhwcTransformer modified: 0 with status: OK
GraphTransformer ConvAddActivationFusion modified: 0 with status: OK
GraphTransformer RemoveDuplicateCastTransformer modified: 0 with status: OK
GraphTransformer CastFloat16Transformer modified: 0 with status: OK
GraphTransformer MemcpyTransformer modified: 0 with status: OK
Node placements
All nodes placed on [TensorrtExecutionProvider]. Number of nodes: 1
SaveMLValueNameIndexMapping
Done saving OrtValue mappings.
Use DeviceBasedPartition as default
Saving initialized tensors.
Done saving initialized tensors
Session successfully initialized.

### To reproduce

Possibly you haven't noticed the timing and it is trivial to recreate. Possibly it has to do with the loaded network's settable sizes, which should be easy to test, or just document the rules for when caching can't work, and why, Possibly it has to do with the particulars of our network structure, but I strongly feel that the graph transformer should have complained in this case.

### Urgency

Out deadline is getting frightfully close. When we started integrating ONNX Runtime in June of 2022 we had no idea it would take this much time. You must realize it is super hard to use when you don't know which hardware you're going to run on.

Now our deadline is in 24Q1 so I'm getting really nervous as the only thing we have up and running on GPU is Wndows/DirectML. Our measurements using Julia integration shows that we can quadruple that performance with TensorRT on NVIDIA but it is soooo hard to get anything working when you have to build from source.

I think this is mostly a documentation issue. For instance there is about 1 line of documentation per trt_ option and even if I go back to TensorRT documentation it turns out it is not a 1:1 mapping. This ulra-brief description uses lots of unexplained concepts such as in this case "profile" that you may have picked up from TensorRT documentation but which would naturally be unknown to _your_ users. The idea of ONNX Runtime is hopefully that you don't have to be an expert on the libraries underlying all providers, but this is basically the situation right now.

### Platform

Windows

### OS Version

11

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

1.16

### ONNX Runtime API

C++

### Architecture

X64

### Execution Provider

TensorRT

### Execution Provider Library Version

TensorRT 6.8.1.6 on CUDA 11.8