microsoft / microsoft/onnxruntime

[Feature Request] Reliable runtime detection of usable Execution Providers (e.g. CUDA)

Open
#27,177 1 comment 1 reaction 0 assignees View on GitHub
ep:CUDA feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

ONNX Runtime exposes `onnxruntime.get_available_providers()` to query available execution providers. However, this API reports **compile-time–enabled providers**, not providers that are **actually usable at runtime**.

As a result, `CUDAExecutionProvider` (and `TensorrtExecutionProvider`) may be reported as available even when required runtime dependencies (e.g. CUDA, cuDNN, or specific shared libraries like `libcublasLt.so`) are missing or incompatible. Attempting to use these providers then fails during session creation or silently falls back to CPU.

This behavior makes it difficult for applications, especially consumer-facing tools, to reliably select the best execution provider or present accurate diagnostics to users.

### Proposed feature

Add a new API (or extend existing APIs) that allows applications to **reliably detect whether an execution provider is usable at runtime**, not just compiled in.

Examples of possible APIs:

```python
onnxruntime.is_provider_usable("CUDAExecutionProvider")
```

or

```python
onnxruntime.get_loadable_providers()
```

Such an API would:

* Attempt to load the provider’s shared library
* Validate required runtime dependencies (e.g. CUDA / cuDNN)
* Return a deterministic result without requiring model loading or exception-based control flow

### Describe scenario use case

This feature would benefit applications that need to **automatically select the optimal execution provider** at startup, including:

* Consumer-facing ML inference applications
* GUI tools that want to display accurate GPU availability status
* Cross-platform software distributed to users with varying CUDA setups
* Applications that want to avoid silent CPU fallbacks or confusing runtime errors

Currently, developers must attempt to create an `InferenceSession` with `CUDAExecutionProvider` and catch exceptions to determine usability. This approach:

* Requires a valid model
* Relies on exceptions for normal control flow
* Produces poor UX when CUDA appears “available” but fails at runtime

A first-class, runtime-accurate provider detection API would simplify application logic, reduce user confusion, and improve robustness.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing onnxruntime.get_available_providers() and the InferenceSession provider-selection flow described in the issue. Compare the proposed is_provider_usable() and get_loadable_providers() APIs, including shared-library and dependency validation requirements. Done means applications can deterministically detect runtime-usable providers without loading a model or using exceptions for normal control flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.