microsoft / microsoft/onnxruntime

Ort::ThrowOnError doesn't throw but crashes due to what seems to be nullptr access

Open
#15,982 9 comments 0 reactions 1 assignee Claimed by @yf711 View on GitHub
ep:CUDA ep:TensorRT platform:windows
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

When I changed GPU in my machine and started my application which tried to load the cached .engine-file for tensorRT execution provider it obviously failed due to wrong compute compability and presented the error message:

`onnxruntime::TensorrtLogger::log] [2023-05-15 09:17:34 ERROR] 6: The engine plan file is generated on an incompatible device, expecting compute 8.6 got compute 7.5, please rebuild.
2023-05-15 09:17:34.8002971 [E:onnxruntime:SparseInst, tensorrt_execution_provider.h:58 onnxruntime::TensorrtLogger::log] [2023-05-15 09:17:34 ERROR] 4: [runtime.cpp::nvinfer1::Runtime::deserializeCudaEngine::66] Error Code 4: Internal Error (Engine deserialization failed.)`

What is weird is that my application didn't catch this error, despite the failing Ort::Session::Run being in a try-catch block but instead crashed.

If I force an error in Ort::Session::Run(...) by using an incompatible .engine file it crashes by raising an unhandled exception:

`Unhandled exception at 0x00007FFFD4FAD2F9 (onnxruntime_providers_tensorrt.dll) in cpp_inference.exe: 0xC0000005: Access violation reading location 0x0000000000000008.`

### To reproduce

Force an error in Ort::Session::Run by loading an incompatible cached TensorRT .engine file (suggestively one built on another machine with different hardware) and try to catch the error. Here's a complete snippet of the code that I could reproduce the issue with (I put the incompatible cached .engine file under C:/tmp/):

```cpp
#include
#include
#include
#include
#include

int main( )
{
constexpr int frameChannels = 3;
constexpr int frameHeight = 640;
constexpr int frameWidth = 640;

const std::string engineCachePath = "C:/tmp/";

std::filesystem::path modelPath = __FILE__;
modelPath.remove_filename( ).append( "model.onnx" );

Ort::Env env( OrtLoggingLevel::ORT_LOGGING_LEVEL_WARNING, "test" );
Ort::SessionOptions sessionOptions;

auto& ortApi = Ort::GetApi( );

// ########## Setup CUDA options ############
OrtCUDAProviderOptionsV2* pCudaOptions = nullptr;
ortApi.CreateCUDAProviderOptions( &pCudaOptions );
std::unique_ptr cudaOptions(
pCudaOptions, ortApi.ReleaseCUDAProviderOptions
);
std::vector keys{ "device_id", "cudnn_conv_use_max_workspace", "do_copy_in_default_stream" };
std::vector values{ "0", "0", "1" };
ortApi.UpdateCUDAProviderOptions( cudaOptions.get( ), keys.data( ), values.data( ), keys.size( ) );
ortApi.SessionOptionsAppendExecutionProvider_CUDA_V2( sessionOptions, cudaOptions.get( ) );

// ########## Setup TRT options ############
OrtTensorRTProviderOptionsV2* pTrtOptions = nullptr;
ortApi.CreateTensorRTProviderOptions( &pTrtOptions );
std::unique_ptr trtOptions(
pTrtOptions, ortApi.ReleaseTensorRTProviderOptions
);
std::vector trtKeys{
"device_id",
"trt_fp16_enable",
"trt_dla_enable",
"trt_dla_core",
"trt_engine_cache_enable",
"trt_engine_cache_path" };
std::vector trtValues{ "0", "1", "1", "0", "1", engineCachePath.c_str( ) };
ortApi.UpdateTensorRTProviderOptions( trtOptions.get( ), trtKeys.data( ), trtValues.data( ), trtKeys.size( ) );
ortApi.SessionOptionsAppendExecutionProvider_TensorRT_V2( sessionOptions, trtOptions.get( ) );

// ###########################################
Ort::Session session{ nullptr };
size_t numInputNodes;
size_t numOutputNodes;
std::vector inputNodeNamesAllocated;
std::vector inputNodeNames;
std::vector outputNodeNamesAllocated;
std::vector outputNodeNames;
std::vector inputTensorShape;
try {
session = Ort::Session( env, modelPath.wstring( ).c_str( ), sessionOptions );

Ort::AllocatorWithDefaultOptions allocator;
numInputNodes = session.GetInputCount( );
for ( size_t idx = 0; idx < numInputNodes; ++idx ) {
inputNodeNamesAllocated.push_back( session.GetInputNameAllocated( idx, allocator ) );
inputNodeNames.push_back( inputNodeNamesAllocated.back( ).get( ) );
}

numOutputNodes = session.GetOutputCount( );
for ( size_t idx = 0; idx < numOutputNodes; ++idx ) {
outputNodeNamesAllocated.push_back( session.GetOutputNameAllocated( idx, allocator ) );
outputNodeNames.push_back( outputNodeNamesAllocated.back( ).get( ) );
}

inputTensorShape = session.GetInputTypeInfo( 0 ).GetTensorTypeAndShapeInfo( ).GetShape( );
for ( auto& dim : inputTensorShape ) {
if ( dim == -1 )
dim = 1;
}
}
catch ( const std::exception& e ) {
std::cout << "Error: " << e.what( ) << std::endl;
return 1;
}

std::unique_ptr dummyImage = std::make_unique( frameWidth * frameHeight * frameChannels );
Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu( OrtDeviceAllocator, OrtMemTypeDefault );
Ort::Value inputTensor = Ort::Value::CreateTensor(
memoryInfo,
dummyImage.get( ),
frameWidth * frameHeight * frameChannels,
inputTensorShape.data( ),
inputTensorShape.size( )
);

Ort::RunOptions runOptions;
try {
std::vector outputTensors = session.Run(
runOptions, inputNodeNames.data( ), &inputTensor, numInputNodes, outputNodeNames.data( ), numOutputNodes
);
}
catch ( const std::exception& e ) {
std::cout << "Error: " << e.what( ) << std::endl;
return 1;
}

return 0;
}
```

### Urgency

_No response_

### Platform

Windows

### OS Version

11

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.13.1 & 1.14.1

### ONNX Runtime API

C++

### Architecture

X64

### Execution Provider

CUDA, TensorRT

### Execution Provider Library Version

CUDA 11.6.2, TensorRT 8.5.1.7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.