microsoft / microsoft/onnxruntime

TensorRT Execution Provider build fail while using TensorRT successfully create engine file

Open
#17,312 4 comments 0 reactions 1 assignee Claimed by @yf711 View on GitHub
ep:CUDA ep:TensorRT platform:windows
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

Hi, I am using ONNX Runtime with TensorRT Execution Provider for a quantized model (YOLO-NAS). While TensorRT cli (trtexec.exe) successfully build the engine from onnx model, the ONNX Runtime with TensorRT Execution Provider cannot build the engine file. Here is the output of TensorRT Execution Provider:

- 2023-08-28 14:32:23.7685241 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 onnxruntime::TensorrtLogger::log] [2023-08-28 07:32:23 WARNING] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading

- 2023-08-28 14:32:29.8989822 [W:onnxruntime:Default, tensorrt_execution_provider.h:75 onnxruntime::TensorrtLogger::log] [2023-08-28 07:32:29 WARNING] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading

- 2023-08-28 14:32:29.9544855 [W:onnxruntime:Default, tensorrt_execution_provider.cc:1934 onnxruntime::TensorrtExecutionProvider::Compile] [TensorRT EP] Try to use DLA core, but platform doesn't have any DLA core

- 2023-08-28 14:32:30.0863651 [E:onnxruntime:Default, tensorrt_execution_provider.h:73 onnxruntime::TensorrtLogger::log] [2023-08-28 07:32:30 ERROR] 2: [graphOptimizer.cpp::nvinfer1::builder::`anonymous-namespace'::chooseHigherPrecision::7174] Error Code 2: Internal Error (Assertion dt1 == DataType::kFLOAT || dt1 == DataType::kHALF failed. )

### To reproduce

ort_session = onnxruntime.InferenceSession(
"model.onnx",
providers=['TensorrtExecutionProvider'],
provider_options=[{'device_id': '0',
'trt_int8_enable':True}] )

link to download model: [https://drive.google.com/file/d/1ZxT2wCU0bIjYrKmhDuFgKRew1kR9hbRd/view?usp=sharing]

### Urgency

Very Urgent

### Platform

Windows

### OS Version

11

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.15.1

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

TensorRT

### Execution Provider Library Version

CUDA 11.6, CUDNN 8.9.0, TENSORRT 8.6.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.