microsoft / microsoft/onnxruntime
[Feature Request] Support building Python-API for multiple ExecutionProviders
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Currently, apparently there is no "official" support for building onnxruntime with multiple ExecutionProviders, see also https://github.com/microsoft/onnxruntime/issues/18790.
The issue I linked to suggests building onnxruntime from scratch as a resolution, and I did exactly that.
However, there were several places in the (python) code where it's hard-coded/assumed that there is only one `ExecutionProvider` available. Two most prominent places that come to mind (and should be considered bugs if onnxruntime says it supports using mulitple ExecutionProviders in parallel):
1. During building for example, package name and EP-flags such as `is_openvino` are mutually exclusive while they do not have to be, see e.g. https://github.com/microsoft/onnxruntime/blob/main/setup.py#L62-L88
2. The error detection and fallback logic of `InferenceSession` are also assuming mutually exclusive `ExecutionProviders`, see for example here https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/onnxruntime_inference_collection.py#L437-L460. This lead to the inability of falling back from `OpenVINOExeuctionProvider` to `CPUExecutionProvider` when the available provider list is `["TensorrtExecutionProvider", "CUDAExecutionProvider", "OpenVINOExecutionProvider", "CPUExecutionProvider"]`, which is the case when compiling with trt, cuda & openvino.
Both of the above should be considered bugs that need to be solved if onnxruntime decides to actually support multiple `ExecutionProviders` officially
### Describe scenario use case
Deployment inside docker where one wants to leverage multiple EPs/different hardware sets, see also https://github.com/microsoft/onnxruntime/issues/18790
Contributor guide
Research direction
Start with setup.py lines 62-88 and onnxruntime/python/onnxruntime_inference_collection.py lines 437-460, then review the linked issue 18790 for the intended multi-ExecutionProvider support. Check how package naming, EP flags, and InferenceSession fallback behave with TensorRT, CUDA, OpenVINO, and CPU together. Done means the Python API can build with multiple providers and fall back from OpenVINOExecutionProvider to CPUExecutionProvider.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100