trtexec failed with cuda driver 570

Description

Environment

TensorRT Version: 10.10

NVIDIA GPU: RTX3090

NVIDIA Driver Version: NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8

CUDA Version: 12.4

CUDNN Version:

Operating System: ubuntu

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Steps To Reproduce

Commands or scripts:

[11/07/2025-06:59:06] [I] === Device Information ===
[11/07/2025-06:59:06] [I] Available Devices:
[11/07/2025-06:59:06] [I] Device 0: "NVIDIA GeForce RTX 3090" UUID: GPU-9196782c-2fae-91c7-c24f-e9a05e4369a7
[11/07/2025-06:59:06] [I] Device 1: "NVIDIA GeForce RTX 3090" UUID: GPU-fa724978-a973-65f4-7a78-26d847638556
[11/07/2025-06:59:06] [I] Selected Device: NVIDIA GeForce RTX 3090
[11/07/2025-06:59:06] [I] Selected Device ID: 0
[11/07/2025-06:59:06] [I] Selected Device UUID: GPU-9196782c-2fae-91c7-c24f-e9a05e4369a7
[11/07/2025-06:59:06] [I] Compute Capability: 8.6
[11/07/2025-06:59:06] [I] SMs: 82
[11/07/2025-06:59:06] [I] Device Global Memory: 24135 MiB
[11/07/2025-06:59:06] [I] Shared Memory per SM: 100 KiB
[11/07/2025-06:59:06] [I] Memory Bus Width: 384 bits (ECC disabled)
[11/07/2025-06:59:06] [I] Application Compute Clock Rate: 1.695 GHz
[11/07/2025-06:59:06] [I] Application Memory Clock Rate: 9.751 GHz
[11/07/2025-06:59:06] [I]
[11/07/2025-06:59:06] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[11/07/2025-06:59:06] [I]
[11/07/2025-06:59:06] [I] TensorRT version: 10.10.0
[11/07/2025-06:59:06] [I] Loading standard plugins
[11/07/2025-06:59:10] [W] [TRT] Unable to determine GPU memory usage: In getGpuMemStatsInBytes at /_src/common/extended/resources.cpp:1175
[11/07/2025-06:59:10] [E] [TRT] createInferBuilder: Error Code 6: API Usage Error (CUDA initialization failure with error: 35. Please check your CUDA installation: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html In operator() at /_src/optimizer/api/builder.cpp:1360)
[11/07/2025-06:59:10] [E] [TRT] [checkMacros.cpp::catchCudaError::229] Error Code 1: Cuda Runtime (In catchCudaError at /_src/common/dispatch/checkMacros.cpp:229)
[11/07/2025-06:59:10] [E] Builder creation failed
[11/07/2025-06:59:10] [E] Failed to create engine from model or file.
[11/07/2025-06:59:10] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v101000] [b31] # ./build_10.10/trtexec --onnx=./samples/sampleNamedDimensions/concat_layer.onnx --saveEngine=test.trt

===============================

=== Help ===
--help, -h Print this message
[11/07/2025-07:20:11] [E] Model missing or format not recognized
&&&& FAILED TensorRT.trtexec [TensorRT v101000] [b31] # ./build_10.10/trtexec --version

==============================

/usr/src/tensorrt/bin/trtexec --version
&&&& RUNNING TensorRT.trtexec [TensorRT v101401] [b48] # /usr/src/tensorrt/bin/trtexec --version
Cuda failure at /_src/samples/common/common.h:1038: CUDA driver version is insufficient for CUDA runtime version

Start with the reported trtexec commands and the CUDA failure at /_src/samples/common/common.h:1038. Compare the build_10.10 and /usr/src/tensorrt trtexec environments, including the reported driver and CUDA runtime versions. Done means identifying whether the failure is a supported compatibility issue and documenting a reproducible diagnosis or resolution.