microsoft / microsoft/onnxruntime
[Feature Request] Drive EP conformance coverage from the provider registry and assert every compiled EP is represented
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Follow-up to #28968, which added an EP conformance test suite (`onnxruntime/test/framework/execution_provider_conformance_test.cc`) encoding backend-agnostic invariants every `IExecutionProvider` must satisfy.
Today that suite's coverage is a hand-maintained list inside `GetEpConformanceParams()`, guarded by `#ifdef USE_*`:
```cpp
params.push_back({"Cpu_Arena", [] { return DefaultCpuExecutionProvider(true); }});
params.push_back({"Cpu_NoArena",[] { return DefaultCpuExecutionProvider(false); }});
#ifdef USE_CUDA
params.push_back({"Cuda", [] { return DefaultCudaExecutionProvider(); }});
#endif
// ... DML, WebGPU, XNNPACK
```
So only 5 EPs are covered, and **a newly added EP is silently not conformance-tested** — nothing fails, nothing warns. The invariants are advertised as backend-agnostic, but enforcement is opt-in and easy to forget.
**Proposal:** derive coverage from the existing provider registry, and fail the build's test run when a compiled-in EP has no conformance entry.
Two pieces already exist that make this straightforward:
1. **An enumerable, build-flag-aware registry.** `onnxruntime/core/providers/get_execution_providers.h` exposes `GetAvailableExecutionProviderNames()` — "the names of execution providers available in this build" — backed by a 21-entry table in which every entry is `USE_*`-guarded. It is already included and exercised from the same test binary by `onnxruntime/test/providers/get_execution_providers_test.cc`, so it is linkable from unit tests today.
2. **Uniform, unconditionally-declared factories.** Every `Default*ExecutionProvider()` in `onnxruntime/test/util/include/default_providers.h` is declared without `#ifdef` and returns `nullptr` when its EP is not compiled. That means the `#ifdef USE_*` blocks in `GetEpConformanceParams()` can be dropped entirely: an EP that isn't compiled simply yields `nullptr`, and the affected test skips instead of failing.
Worth noting as corroboration that `GetAvailableExecutionProviderNames()` is the right source of truth: its WebGPU entry is guarded by `#if defined(USE_WEBGPU) && !defined(ORT_USE_EP_API_ADAPTERS)` — identical to the guard #28968 arrived at independently for the same EP.
**Sketch**
Add the canonical EP name to the parameter struct, then assert coverage in one direction:
```cpp
struct EpConformanceParam {
std::string name; // gtest parameter suffix
std::string_view ep_name; // canonical kXxxExecutionProvider
std::function()> factory;
bool expects_plugin_ep = false;
};
// EPs compiled into some builds that intentionally have no conformance entry.
constexpr std::string_view kConformanceExemptEps[] = {
kJsExecutionProvider, // web/emscripten only
kWebNNExecutionProvider, // web/emscripten only
kAzureExecutionProvider, // remote endpoint EP, not a compute EP
kVitisAIExecutionProvider, // requires external config/runtime to construct
};
TEST(EpConformanceCoverage, EveryAvailableEpIsRegistered) {
std::set registered;
for (const auto& p : GetEpConformanceParams()) registered.insert(p.ep_name);
for (const auto& name : GetAvailableExecutionProviderNames()) {
if (IsExempt(name)) continue;
EXPECT_TRUE(registered.count(name))
<< name << " is compiled into this build but has no EP conformance entry. "
<< "Add it to GetEpConformanceParams(), or to kConformanceExemptEps with a justification.";
}
}
```
The assertion is deliberately **one-directional** ("every available EP is registered", not a biconditional) so it does not false-fail on the reverse mismatch — e.g. `DefaultSnpeExecutionProvider()` exists but SNPE is not in the availability table.
**Known gap.** Four EPs appear in the availability table but have no `Default*ExecutionProvider()` helper, so they need either a new factory or a documented exemption:
| EP | Reason it has no factory |
|---|---|
| `kJsExecutionProvider` | web/emscripten-only; not constructible in a native test binary |
| `kWebNNExecutionProvider` | web/emscripten-only |
| `kAzureExecutionProvider` | remote endpoint EP rather than a compute EP |
| `kVitisAIExecutionProvider` | requires external config/runtime to construct |
**Staging / risk.** Expanding coverage means CI legs that compile QNN, OpenVINO, TensorRT, MIGraphX, CANN, CoreML, etc. would begin running the invariants against EPs that have never been checked, which may surface genuine pre-existing violations. To keep that tractable, this is best landed as a ratchet: introduce the coverage assertion with not-yet-vetted EPs seeded into `kConformanceExemptEps`, then graduate them out one at a time as each is validated. That keeps CI green while making the gap explicit and steadily shrinking, rather than turning many legs red at once.
Note that #28968 already makes unavailable EPs skip rather than fail in both directions — a factory returning `nullptr`, and a factory that throws during construction (e.g. CUDA's constructor calling `cudaSetDevice` on a machine with no device) — so widening the list does not by itself introduce environment-dependent failures.
### Describe scenario use case
The EP conformance suite exists so that EP authors — increasingly third parties writing plugin EPs against the public EP ABI — get a mechanical answer to "does my provider actually satisfy the contract the framework relies on?" That guarantee is only as good as its coverage.
Concretely this supports:
- **New EP onboarding.** Someone adding an EP (or a vendor adding an out-of-tree one) is told at test time that their provider is unchecked, instead of finding out later through a framework-level bug that traces back to a violated `IExecutionProvider` assumption.
- **Preventing silent coverage regressions.** Coverage cannot quietly decay as EPs are added, renamed, or reorganized, because the registry and the conformance list are cross-checked rather than independently maintained.
- **An explicit, reviewable exemption list.** EPs that genuinely cannot be conformance-tested (web-only, remote-endpoint, external-runtime) are enumerated with justifications in code, so "not covered" becomes a deliberate, reviewed decision rather than an accident of omission.
- **Less boilerplate.** Dropping the per-EP `#ifdef USE_*` blocks removes a class of copy-paste error in the test list itself.
Happy to put up the PR if this direction sounds right.
Contributor guide
Assessment
This issue has not been assessed yet.