microsoft / microsoft/onnxruntime

[Documentation] how to modularize ONNXRT on CPU first , then on CPU with OpenVino EP then on Nvidia GPU with TRT EP simply by adding new provider libraries and all their dependencies

Open
#23,104 4 comments 0 reactions 0 assignees View on GitHub
documentation ep:OpenVINO ep:TensorRT
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the documentation issue

The goal is to create modular deployment profiles (such as docker image layers) that you pile up when you add the provider capability . See the diagram below:
![Image](https://github.com/user-attachments/assets/d9f708f2-74e0-4ba4-ae83-d0b97c39a893)

This is supposed to work already like said in [Build with different EPs - onnxruntime](https://chenfucn.github.io/onnxruntime/docs/build/eps.html#execution-provider-shared-libraries)
Quoting :
"Execution Provider Shared Libraries

The TensorRT, and OpenVINO™ providers are built as shared libraries vs being statically linked into the main onnxruntime. This enables them to be loaded only when needed, and if the dependent libraries of the provider are not installed onnxruntime will still run fine, it just will not be able to use that provider.

Loading the shared providers

Shared provider libraries are loaded by the onnxruntime code (do not load or depend on them in your client code…. [libraries] will be loaded at runtime when the provider is added to the session options (through a call like SessionOptionsAppendExecutionProvider_OpenVINO in the C API). If a shared provider library cannot be loaded (if the file doesn’t exist, or its dependencies don’t exist or not in the path) then an error will be returned.

The onnxruntime code will look for the provider shared libraries in the same location as the onnxruntime shared library is (or the executable statically linked to the static library version)."

However , it does not seems to work because when building ONNXRT with TRT EP , I am getting both CUDA EP and TRT EP, but , if I remove the libonnxruntime_providers_cuda.so, then my Client code does got runtime link error looking for this library despite my client code does not add (depend) on OrtSessionOptionsAppendExecutionProvider_CUDA. See [[Build] Cuda Execution Provider library is needed despite we only use TensoRT Execution provider · Issue #22960 · microsoft/onnxruntime](https://github.com/microsoft/onnxruntime/issues/22960)

Here is how I built ONNRT :
CC=gcc-11 CXX=g++-11 ./build.sh \
--skip_submodule_sync --nvcc_threads 2 \
--config $ORT_BUILD_MODE --use_cuda \
--cudnn_home /usr/local/cuda/lib64 \
--cuda_home /usr/local/cuda/
--use_tensorrt --use_tensorrt_oss_parser --tensorrt_home /usr/local/TensorRT \
--build_shared_lib --parallel --skip_tests \
--allow_running_as_root --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=75" \
--cmake_extra_defines "CMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-11"

So can you tell how to build all provider libs and deploy them separately based on my CPU / GPU host configuration ?
Onnxruntime shall run even if a provider library is not there as the client code does not use it.

Shall I use all provider in the build such as ?
CC=gcc-11 CXX=g++-11 ./build.sh \
--skip_submodule_sync --nvcc_threads 2 \
--config $ORT_BUILD_MODE --use_cuda \
--cudnn_home /usr/local/cuda/lib64 \
--cuda_home /usr/local/cuda/
--use_tensorrt --use_tensorrt_oss_parser --tensorrt_home /usr/local/TensorRT \
--use_openvino --openvino_home /usr/local/OpenVino \
--build_shared_lib --parallel --skip_tests \
--allow_running_as_root --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=75" \
--cmake_extra_defines "CMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-11"

Please advise

### Page / URL

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with docs/build/eps.html and the linked issue #22960, then reproduce the reported build using build.sh and the CUDA, TensorRT, and OpenVINO options shown here. Done means the documentation explains how to build and deploy provider libraries separately, including the expected behavior when a provider library or dependency is absent.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
build-system, documentation, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.