dymamic shape model illegal memory error

Description

i use tensorrt8.5.3 convert my model,but my model have dynamic shape and dynamic batch,So I made relevant configuration，but encouter illegal memory error：
Debug:Context start enqueuev2*****
[2023-05-04 06:28:29 ERROR] 1: [deviceToShapeHostRunner.cpp::execute::39] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
cudaEngineCheck(CH=0,Status=-2) failed at src/onnx2trt_lib.cpp:1610 : an illegal memory access was encountered
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
CH.0 Average over 5 runs is 12.9119 ms.
CUDA MemInfo total = 0Bytes, free = 0Bytes, Delta = 4579328000Bytes
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
CH.0 Average over 5 runs is 12.3975 ms.
CUDA MemInfo total = 0Bytes, free = 0Bytes, Delta = 0Bytes
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
[2023-05-04 06:28:29 ERROR] 1: [context.cpp::setStream::82] Error Code 1: Cudnn (CUDNN_STATUS_MAPPING_ERROR)
CH.0 Average over 5 runs is 11.985 ms.
CUDA MemInfo total = 0Bytes, free = 0Bytes, Delta = 0Bytes
cudaEngineCheck(CH=0,Status=-2) failed at src/onnx2trt_lib.cpp:1638 : an illegal memory access was encountered
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5592ed1ce210'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5592ed1e5bf0'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5593059ac570'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5592ed1a64a0'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5592ed230f80'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5593064c32c0'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x559306b375b0'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x559305c40de0'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5593060c8020'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x55930532f040'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x559305ad1f70'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x55930704e820'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x55930511f950'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x559305c07fd0'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5593052a4f30'.)
[2023-05-04 06:28:29 ERROR] 1: [graphContext.h::~MyelinGraphContext::35] Error Code 1: Myelin (Error 700 destroying stream '0x5593060d1c80'.)
[2023-05-04 06:28:29 ERROR] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[2023-05-04 06:28:29 ERROR] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)

There are some warnings before reporting an error:

[2023-05-04 06:24:56 WARNING] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.2.6
[2023-05-04 06:27:48 WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: [verify_output_type] Mismatched type for tensor (Unnamed Layer_ 154) [Shuffle]output', f16 vs. expected type:f32.
[2023-05-04 06:27:48 WARNING] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[2023-05-04 06:27:48 WARNING] (# 0 (VALUE NonZero_172[size]))
[2023-05-04 06:27:48 WARNING] (# 0 (VALUE NonZero_179[size]))
[2023-05-04 06:27:50 WARNING] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[2023-05-04 06:27:50 WARNING] (# 0 (VALUE NonZero_172[size]))
[2023-05-04 06:27:50 WARNING] (# 0 (VALUE NonZero_179[size]))
[2023-05-04 06:27:57 WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: [verify_output_type] Mismatched type for tensor (Unnamed Layer 318) [Shuffle]output', f16 vs. expected type:f32.
[2023-05-04 06:28:17 WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: [verify_output_type] Mismatched type for tensor (Unnamed Layer 499) [Shuffle]output', f16 vs. expected type:f32.
[2023-05-04 06:28:28 WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: [verify_output_type] Mismatched type for tensor (Unnamed Layer 518) [Shuffle]_output', f16 vs. expected type:f32.
[2023-05-04 06:28:28 WARNING] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[2023-05-04 06:28:28 WARNING] (# 0 (VALUE NonZero_172[size]))
[2023-05-04 06:28:28 WARNING] (# 0 (VALUE NonZero_179[size]))
[2023-05-04 06:28:28 WARNING] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.2.6
[2023-05-04 06:28:28 WARNING] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[2023-05-04 06:28:28 WARNING] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[2023-05-04 06:28:28 WARNING] Check verbose logs for the list of affected weights.
[2023-05-04 06:28:28 WARNING] - 24 weights are affected by this issue: Detected subnormal FP16 values.
[2023-05-04 06:28:28 WARNING] - 3 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.

I would like to ask what is the general reason for this kind of problem, or what tools or methods are generally used to solve， If you need a specific model, I can provide it here, thank you very much

Environment

TensorRT Version: 8.5.3
NVIDIA GPU: P2200
NVIDIA Driver Version:
CUDA Version: 11.4
CUDNN Version:
Operating System: ubantu20.04
Python Version (if applicable):
Tensorflow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if so, version):

Relevant Files

Steps To Reproduce

Start with the failure reported from src/onnx2trt_lib.cpp and the related entries in deviceToShapeHostRunner.cpp, context.cpp, graphContext.h, defaultAllocator.cpp, and cudaResources.cpp. Obtain the missing model, reproduction commands, driver and cuDNN versions, then reduce the dynamic-shape case until the illegal memory access is reproducible and its cause can be verified.