DCHECK(mem_span <= data_->size()) failed. mem_span: 2559986, data_->size()320000

Description

[09/19/2024-13:28:56] [V] [TRT] --------------- Timing Runner: /img_backbone/layer2/layer2.2/conv1/Conv + /img_backbone/layer2/layer2.2/relu/Relu (CublasConvolution)
[09/19/2024-13:28:56] [V] [TRT] CublasConvolution has no valid tactics for this config, skipping
[09/19/2024-13:28:56] [V] [TRT] =============== Computing costs for 
[09/19/2024-13:28:56] [V] [TRT] *************** Autotuning format combination: Half(30,5,1), Half(16,1:8,4,1), Half(16,1:8,4,1), Half(16,1:8,4,1), Half(1280000,1:8,160000,5000), Half(10240000,256,1) -> Half(10240000,256,1), Half(10240000,256,1), Half(10240000,256,1), Half(160000,4,1:8,4,1), Half(80000,2,2,1:8,2,1), Half(2560000,64,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(320000,8,8,8,1:8,2,1), Half(10240000,256,1), Half(10240000,256,1), Half(5120000,128,16,1:8,2,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1:8,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1), Half(40000,1,1) ***************
[09/19/2024-13:28:56] [V] [TRT] --------------- Timing Runner: {ForeignNode[onnx::Expand_1636.../pts_bbox_head/encoder/layers.0/attentions.1/Cast_3]} (Myelin)
operand.cpp:61: DCHECK(mem_span <= data_->size()) failed. mem_span: 2559986, data_->size()320000
Segmentation fault (core dumped)   

./trtexec --onnx=./trt_bev/bevf-864-1536-200-200_trt_plugin_fixed_20240919_prev_bev_opt_15.onnx   \
--fp16   --dumpProfile  \
--saveEngine=./trt8510_fp16.plan \
--tacticSources=-CUDNN,-CUBLAS,-CUBLAS_LT \
--workspace=4096 --verbose

Environment

TensorRT Version:8.5.10 Drive os 6060

NVIDIA GPU: Drive agx orin-x

NVIDIA Driver Version:

CUDA Version:11.4

CUDNN Version:11.6

I think this is bug of trt8.5.10 (limit with drive os version).
@ttyio Can you help confirm it ?

fp32 passsed. fp16 wrong.

Reproduce the failure with the provided trtexec command, TensorRT 8.5.10, Drive OS 6060, and FP16 settings, then compare it with the reported FP32 success. Start at operand.cpp:61 and inspect the failing Myelin path around the reported memory-span assertion. Done means determining whether the failure is a reproducible TensorRT bug or an environment/model issue, with supporting logs or a validated fix.