microsoft / microsoft/onnxruntime
[Feature Request] Reliable runtime detection of usable Execution Providers (e.g. CUDA)
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
ONNX Runtime exposes `onnxruntime.get_available_providers()` to query available execution providers. However, this API reports **compile-time–enabled providers**, not providers that are **actually usable at runtime**.
As a result, `CUDAExecutionProvider` (and `TensorrtExecutionProvider`) may be reported as available even when required runtime dependencies (e.g. CUDA, cuDNN, or specific shared libraries like `libcublasLt.so`) are missing or incompatible. Attempting to use these providers then fails during session creation or silently falls back to CPU.
This behavior makes it difficult for applications, especially consumer-facing tools, to reliably select the best execution provider or present accurate diagnostics to users.
### Proposed feature
Add a new API (or extend existing APIs) that allows applications to **reliably detect whether an execution provider is usable at runtime**, not just compiled in.
Examples of possible APIs:
```python
onnxruntime.is_provider_usable("CUDAExecutionProvider")
```
or
```python
onnxruntime.get_loadable_providers()
```
Such an API would:
* Attempt to load the provider’s shared library
* Validate required runtime dependencies (e.g. CUDA / cuDNN)
* Return a deterministic result without requiring model loading or exception-based control flow
### Describe scenario use case
This feature would benefit applications that need to **automatically select the optimal execution provider** at startup, including:
* Consumer-facing ML inference applications
* GUI tools that want to display accurate GPU availability status
* Cross-platform software distributed to users with varying CUDA setups
* Applications that want to avoid silent CPU fallbacks or confusing runtime errors
Currently, developers must attempt to create an `InferenceSession` with `CUDAExecutionProvider` and catch exceptions to determine usability. This approach:
* Requires a valid model
* Relies on exceptions for normal control flow
* Produces poor UX when CUDA appears “available” but fails at runtime
A first-class, runtime-accurate provider detection API would simplify application logic, reduce user confusion, and improve robustness.
Contributor guide
Research direction
Start by reviewing onnxruntime.get_available_providers() and the InferenceSession provider-selection flow described in the issue. Compare the proposed is_provider_usable() and get_loadable_providers() APIs, including shared-library and dependency validation requirements. Done means applications can deterministically detect runtime-usable providers without loading a model or using exceptions for normal control flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend-api-design, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100