microsoft / microsoft/onnxruntime

[Feature Request] Support the WebGPU EP as a fallback for the OpenVINO NPU

Open
#32,625 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

ep:OpenVINO ep:WebGPU ep:WebNN feature request platform:web
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

### Background

The OpenVINO EP's NPU device has limited operator coverage — a restricted set of data types and tensor ranks. Previously, when running under WebNN, subgraphs the NPU could not handle fell back to the OpenVINO CPU device or the default ORT CPU EP, so a model the NPU could not fully execute still had a path to run.

With the recent WebNN Compiler process re-architecture, CPU fallback has been disabled for security reasons. As a consequence, if any part of a model is unsupported by the OV NPU, compilation now fails outright, which is a poor end-user experience.

### Proposal

We would like to explore using the **WebGPU EP as the new fallback path**, replacing the CPU EP, for subgraphs the OV NPU cannot execute. The WebGPU EP has substantially broader operator and data-type coverage than the NPU, so it can absorb most of the currently-failing subgraphs while keeping execution off the CPU. The goal is to *expand* the supported surface, not to guarantee 100% coverage — a node unsupported by both the NPU and WebGPU would still fail, which we consider acceptable.

This request is to initiate the design discussion and collect feedback.

### Considerations / open questions

1. **Fallback granularity — minimal fragmentation.** The OV EP over-claims at `GetCapability` (it declares support broadly), so ORT hands it a large contiguous subgraph, and whether the NPU can actually compile it is only determined at compile time. This means the fallback unit is a whole OV-claimed subgraph rather than per-operator interleaving, so cross-EP boundaries — and therefore copy overhead — stay few.

2. **Cross-EP tensor transfer is manageable.** The OV EP now registers host-accessible (remote-tensor) memory, so a boundary between OV and WebGPU is effectively "CPU-accessible ↔ WebGPU" rather than "vendor device ↔ WebGPU". This avoids WebGPU rejecting foreign vendor memory and removes the NPU-side device copy. The remaining cost is the WebGPU upload/download, which is incurred at inference runtime (not just at compile time), but given the coarse partitioning above it is typically one hop into and one hop out of the WebGPU subgraph — bounded and acceptable.

3. **Fits the offline-compile architecture.** WebGPU offline compilation is essentially a graph transformation and needs no real GPU adapter, so both the fallback decision and the WebGPU-side offline compile can run inside the device-free compiler process; the actual shader compilation and execution are deferred to the runtime GPU process as usual. WebGPU needing a device is therefore not a blocker at compile time.

4. **Open — where the fallback orchestration lives.** Today's OV→CPU fallback is internal to the OV EP (it recompiles the same subgraph on the OV CPU device). Falling back to WebGPU crosses an EP boundary — the OV EP cannot invoke another EP in-place — so the orchestration needs an explicit owner. Two natural shapes to evaluate: (a) a two-pass / re-partition flow where the failed nodes are re-assigned to the WebGPU EP; or (b) the OV EP emits only the NPU-supported part as EPContext and leaves the failed subgraph as plain ONNX nodes for a downstream WebGPU offline-compile pass. Option (b) aligns well with the existing "emit optimized plain model" path but requires the OV EP to deliver compilation for only part of the nodes it claimed.

Feedback on any of the above — and on whether the WebGPU EP is the right fallback target — is very welcome. Thanks!

@adrastogi @skottmckay @huningxin @ibelem @fdwr @RafaelCintron

### Describe scenario use case

WebNN on Web browser.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by tracing the existing OV-to-CPU fallback and the WebGPU offline-compile path described in the issue, then compare the two proposed orchestration shapes. Done means the fallback owner, supported scope, and failure behavior are agreed before implementation begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents, compilers, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.