microsoft / microsoft/onnxruntime
Ort::ThrowOnError doesn't throw but crashes due to what seems to be nullptr access
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
When I changed GPU in my machine and started my application which tried to load the cached .engine-file for tensorRT execution provider it obviously failed due to wrong compute compability and presented the error message:
`onnxruntime::TensorrtLogger::log] [2023-05-15 09:17:34 ERROR] 6: The engine plan file is generated on an incompatible device, expecting compute 8.6 got compute 7.5, please rebuild.
2023-05-15 09:17:34.8002971 [E:onnxruntime:SparseInst, tensorrt_execution_provider.h:58 onnxruntime::TensorrtLogger::log] [2023-05-15 09:17:34 ERROR] 4: [runtime.cpp::nvinfer1::Runtime::deserializeCudaEngine::66] Error Code 4: Internal Error (Engine deserialization failed.)`
What is weird is that my application didn't catch this error, despite the failing Ort::Session::Run being in a try-catch block but instead crashed.
If I force an error in Ort::Session::Run(...) by using an incompatible .engine file it crashes by raising an unhandled exception:
`Unhandled exception at 0x00007FFFD4FAD2F9 (onnxruntime_providers_tensorrt.dll) in cpp_inference.exe: 0xC0000005: Access violation reading location 0x0000000000000008.`
### To reproduce
Force an error in Ort::Session::Run by loading an incompatible cached TensorRT .engine file (suggestively one built on another machine with different hardware) and try to catch the error. Here's a complete snippet of the code that I could reproduce the issue with (I put the incompatible cached .engine file under C:/tmp/):
```cpp
#include
#include
#include
#include
#include
int main( )
{
constexpr int frameChannels = 3;
constexpr int frameHeight = 640;
constexpr int frameWidth = 640;
const std::string engineCachePath = "C:/tmp/";
std::filesystem::path modelPath = __FILE__;
modelPath.remove_filename( ).append( "model.onnx" );
Ort::Env env( OrtLoggingLevel::ORT_LOGGING_LEVEL_WARNING, "test" );
Ort::SessionOptions sessionOptions;
auto& ortApi = Ort::GetApi( );
// ########## Setup CUDA options ############
OrtCUDAProviderOptionsV2* pCudaOptions = nullptr;
ortApi.CreateCUDAProviderOptions( &pCudaOptions );
std::unique_ptr cudaOptions(
pCudaOptions, ortApi.ReleaseCUDAProviderOptions
);
std::vector keys{ "device_id", "cudnn_conv_use_max_workspace", "do_copy_in_default_stream" };
std::vector values{ "0", "0", "1" };
ortApi.UpdateCUDAProviderOptions( cudaOptions.get( ), keys.data( ), values.data( ), keys.size( ) );
ortApi.SessionOptionsAppendExecutionProvider_CUDA_V2( sessionOptions, cudaOptions.get( ) );
// ########## Setup TRT options ############
OrtTensorRTProviderOptionsV2* pTrtOptions = nullptr;
ortApi.CreateTensorRTProviderOptions( &pTrtOptions );
std::unique_ptr trtOptions(
pTrtOptions, ortApi.ReleaseTensorRTProviderOptions
);
std::vector trtKeys{
"device_id",
"trt_fp16_enable",
"trt_dla_enable",
"trt_dla_core",
"trt_engine_cache_enable",
"trt_engine_cache_path" };
std::vector trtValues{ "0", "1", "1", "0", "1", engineCachePath.c_str( ) };
ortApi.UpdateTensorRTProviderOptions( trtOptions.get( ), trtKeys.data( ), trtValues.data( ), trtKeys.size( ) );
ortApi.SessionOptionsAppendExecutionProvider_TensorRT_V2( sessionOptions, trtOptions.get( ) );
// ###########################################
Ort::Session session{ nullptr };
size_t numInputNodes;
size_t numOutputNodes;
std::vector inputNodeNamesAllocated;
std::vector inputNodeNames;
std::vector outputNodeNamesAllocated;
std::vector outputNodeNames;
std::vector inputTensorShape;
try {
session = Ort::Session( env, modelPath.wstring( ).c_str( ), sessionOptions );
Ort::AllocatorWithDefaultOptions allocator;
numInputNodes = session.GetInputCount( );
for ( size_t idx = 0; idx < numInputNodes; ++idx ) {
inputNodeNamesAllocated.push_back( session.GetInputNameAllocated( idx, allocator ) );
inputNodeNames.push_back( inputNodeNamesAllocated.back( ).get( ) );
}
numOutputNodes = session.GetOutputCount( );
for ( size_t idx = 0; idx < numOutputNodes; ++idx ) {
outputNodeNamesAllocated.push_back( session.GetOutputNameAllocated( idx, allocator ) );
outputNodeNames.push_back( outputNodeNamesAllocated.back( ).get( ) );
}
inputTensorShape = session.GetInputTypeInfo( 0 ).GetTensorTypeAndShapeInfo( ).GetShape( );
for ( auto& dim : inputTensorShape ) {
if ( dim == -1 )
dim = 1;
}
}
catch ( const std::exception& e ) {
std::cout << "Error: " << e.what( ) << std::endl;
return 1;
}
std::unique_ptr dummyImage = std::make_unique( frameWidth * frameHeight * frameChannels );
Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu( OrtDeviceAllocator, OrtMemTypeDefault );
Ort::Value inputTensor = Ort::Value::CreateTensor(
memoryInfo,
dummyImage.get( ),
frameWidth * frameHeight * frameChannels,
inputTensorShape.data( ),
inputTensorShape.size( )
);
Ort::RunOptions runOptions;
try {
std::vector outputTensors = session.Run(
runOptions, inputNodeNames.data( ), &inputTensor, numInputNodes, outputNodeNames.data( ), numOutputNodes
);
}
catch ( const std::exception& e ) {
std::cout << "Error: " << e.what( ) << std::endl;
return 1;
}
return 0;
}
```
### Urgency
_No response_
### Platform
Windows
### OS Version
11
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.13.1 & 1.14.1
### ONNX Runtime API
C++
### Architecture
X64
### Execution Provider
CUDA, TensorRT
### Execution Provider Library Version
CUDA 11.6.2, TensorRT 8.5.1.7
Contributor guide
Assessment
This issue has not been assessed yet.