microsoft / microsoft/onnxruntime

TensorRT EP could not open shared library from ?

Open
#11,271 4 comments 1 reaction 2 assignees View on GitHub

@oliviajain is already working on this.

Since Apr 25, 2022.

ep:TensorRT
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Describe the bug
When I was running yolov5 onnx modle on jetson xavier nx with onnxruntime 1.10 version, I encountererd this error.

what():  /home/robin/DataDisk/buildFromSource/onnxruntime/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:554 onnxruntime::TensorrtExecutionProvider::TensorrtExecutionProvider(const onnxruntime::TensorrtExecutionProviderInfo&) [ONNXRuntimeError] : 11 : EP_FAIL : TensorRT EP could not open shared library from �

Urgency
Normal

System information

  • OS Platform and Distribution (Linux Ubuntu 18.04):
  • ONNX Runtime installed from (source or binary): From Source
  • ONNX Runtime version: 1.10
  • GCC/Compiler version (if compiling from source): onnxruntime is build by gcc7.5, my application is build by gcc11.1
  • CUDA/cuDNN version: 10.2
  • GPU model and memory: Jetson Xavier NX with L4t 32.6.1

To Reproduce
I checked out the onnxruntime git tag to 1.10. I don't use the latest 1.11 version because 1.11 needs tensor rt 8.2 but the tensor rt version on jetson is 8.0 at present. Apparently, the 1.11 version cannot be built successfully with the --use_tensorrt flag.
I build onnxruntime from source on jetson xavier nx, the build command is

./build.sh --config Release --update --build --parallel \
 --build_shared_lib -use_tensorrt \
 --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu \
 --tensorrt_home /usr/lib/aarch64-linux-gnu

After building, I copied all lib.so and include directory to a single folder. Then I linked these files manually to build my application.

set(ONNXRUNTIME_INCLUDE_DIR "${ONNX_INSTALL_DIR}/include" "${ONNX_INSTALL_DIR}/include/onnxruntime/core/session")

find_library(ONNXRUNTIME_LIBRARY NAMES onnxruntime 
    HINTS ${ONNX_INSTALL_DIR}
    PATH_SUFFIXES lib64 lib
    PATHS ${ONNX_INSTALL_DIR}
    DOC "ONNXRUNTIME - Library")
find_library(ONNXRUNTIME_TENSORRT NAMES onnxruntime_providers_tensorrt 
    HINTS ${ONNX_INSTALL_DIR}
    PATH_SUFFIXES lib64 lib
    PATHS ${ONNX_INSTALL_DIR}
    DOC "ONNXRUNTIME Tensorrt - Library")

target_include_directories(OnnxDetector PRIVATE ${ONNXRUNTIME_INCLUDE_DIR})
target_link_libraries(OnnxDetector PRIVATE ${ONNXRUNTIME_LIBRARY} ${ONNXRUNTIME_TENSORRT})

The application reference codes and models are comes from this repo
I tried to use tensorrt instead of normal cuda inference.

sessionOptions = Ort::SessionOptions();
std::vector<std::string> availableProviders = Ort::GetAvailableProviders();
auto trtAvailable = std::find(availableProviders.begin(), availableProviders.end(), "TensorrtExecutionProvider");
OrtTensorRTProviderOptions tensorrtOpt;
tensorrtOpt.device_id = 0;
if (device == ONNXDevices::TENSOR_RT && (trtAvailable != availableProviders.end())){
        std::cout << "Inference device: Tensor rt" << std::endl;
        sessionOptions.AppendExecutionProvider_TensorRT(tensorrtOpt);
}

After building, the application show the error as above. Furthermore, this application could under the normal cuda successfully.
Expected behavior
I hope the application can output inference results.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
yolov5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.