microsoft / microsoft/onnxruntime

[Feature Request] Support building Python-API for multiple ExecutionProviders

Open
#19,268 0 comments 2 reactions 0 assignees View on GitHub
ep:CUDA ep:OpenVINO ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Currently, apparently there is no "official" support for building onnxruntime with multiple ExecutionProviders, see also https://github.com/microsoft/onnxruntime/issues/18790.

The issue I linked to suggests building onnxruntime from scratch as a resolution, and I did exactly that.
However, there were several places in the (python) code where it's hard-coded/assumed that there is only one `ExecutionProvider` available. Two most prominent places that come to mind (and should be considered bugs if onnxruntime says it supports using mulitple ExecutionProviders in parallel):

1. During building for example, package name and EP-flags such as `is_openvino` are mutually exclusive while they do not have to be, see e.g. https://github.com/microsoft/onnxruntime/blob/main/setup.py#L62-L88
2. The error detection and fallback logic of `InferenceSession` are also assuming mutually exclusive `ExecutionProviders`, see for example here https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/onnxruntime_inference_collection.py#L437-L460. This lead to the inability of falling back from `OpenVINOExeuctionProvider` to `CPUExecutionProvider` when the available provider list is `["TensorrtExecutionProvider", "CUDAExecutionProvider", "OpenVINOExecutionProvider", "CPUExecutionProvider"]`, which is the case when compiling with trt, cuda & openvino.

Both of the above should be considered bugs that need to be solved if onnxruntime decides to actually support multiple `ExecutionProviders` officially

### Describe scenario use case

Deployment inside docker where one wants to leverage multiple EPs/different hardware sets, see also https://github.com/microsoft/onnxruntime/issues/18790

Contributor guide

Open the contributing guide

Research direction

Start with setup.py lines 62-88 and onnxruntime/python/onnxruntime_inference_collection.py lines 437-460, then review the linked issue 18790 for the intended multi-ExecutionProvider support. Check how package naming, EP flags, and InferenceSession fallback behave with TensorRT, CUDA, OpenVINO, and CPU together. Done means the Python API can build with multiple providers and fall back from OpenVINOExecutionProvider to CPUExecutionProvider.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.