microsoft / microsoft/onnxruntime
[WebGPU] Non-finite output with ORT_ENABLE_EXTENDED graph optimization (minimal 8-node repro)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
The WebGPU EP produces **non-finite values** (`±Inf` → saturating a downstream `Tanh` to `±1`) when `graph_optimization_level` is `ORT_ENABLE_EXTENDED` or higher (the default `ORT_ENABLE_ALL`). The same model and input are correct on the CPU EP, and correct on the WebGPU EP at `ORT_DISABLE_ALL` / `ORT_ENABLE_BASIC`.
The graph is tiny — 8 nodes: `Split`, `Conv` ×3, `Relu` ×2, `Tanh`, `Constant`. Two of the convs are `stride=2` with folded BatchNorm.
```
graph_optimization_level non-finite % max |diff| vs CPU
ORT_DISABLE_ALL 0.000% 2.965e-06 OK
ORT_ENABLE_BASIC 0.000% 2.965e-06 OK
ORT_ENABLE_EXTENDED 0.414% 1.000e+00 BAD
ORT_ENABLE_ALL (default) 0.414% 1.000e+00 BAD
```
`max |diff| = 1.000` is exactly the `Tanh` saturation value, i.e. the value reaching `Tanh` is already `±Inf`.
Since the level that flips the behaviour is `EXTENDED`, this points at the Conv+BatchNorm / Conv+Activation fusion path in the WebGPU EP rather than at any individual kernel.
**Workaround:** `sess_options.graph_optimization_level = ORT_ENABLE_BASIC` — verified on our full model (97 inference runs), which then matches the CPU EP to `p50` relative difference `2.5e-7`.
#### Hypotheses I ruled out (to save you the same search)
1. **Non-power-of-two channel count** (the chain is 20→32→32→20). Ruled out: the identical chain with *random* weights and 20 channels is clean.
2. **Resolution dependence.** Ruled out: identical chain at both `144×192` and `288×384` with random weights is clean.
3. **BatchNorm with near-zero variance.** Ruled out: `running_var` min is 4.07, so `1/sqrt(var+eps)` peaks at 0.50 — no amplification.
A 2×2 isolation shows it needs **both** the real weights **and** the real input:
| weights | input | non-finite |
|---------|--------|-----------|
| real | real | **0.414%** |
| real | random | 0.000% |
| random | real | 0.000% |
| random | random | 0.000% |
Two more data points that may help narrow it down:
- The trigger is **very robust to quantization**: rounding the input to 1 decimal place and storing it as fp16 leaves the non-finite fraction at *exactly* 0.414%.
- A **constant** input (filled with the median value) does **not** trigger it, so spatial structure matters.
Notably, exposing the intermediate tensors as graph outputs (which prevents fusion) also makes the problem disappear — that is what pointed at fusion in the first place.
### To reproduce
All files (self-contained, ~2.3 MB): https://gist.github.com/Kyle-Wang0211/b1b2a011d62a8ef262f79aac6dccc56d
```bash
# the two binaries are base64 in the gist because gists are text-only
bash decode.sh # decodes model.onnx + input_small.npz, then runs repro.py
```
`repro.py`:
```python
import numpy as np
import onnxruntime as ort
x = np.load("input_small.npz")["ctx"].astype(np.float32) # (1, 36, 288, 384)
levels = [
("ORT_DISABLE_ALL", ort.GraphOptimizationLevel.ORT_DISABLE_ALL),
("ORT_ENABLE_BASIC", ort.GraphOptimizationLevel.ORT_ENABLE_BASIC),
("ORT_ENABLE_EXTENDED", ort.GraphOptimizationLevel.ORT_ENABLE_EXTENDED),
("ORT_ENABLE_ALL", ort.GraphOptimizationLevel.ORT_ENABLE_ALL),
]
def run(providers, level):
so = ort.SessionOptions()
so.graph_optimization_level = level
return ort.InferenceSession("model.onnx", so, providers=providers).run(
["hidden"], {"ctx": x})[0]
ref = run(["CPUExecutionProvider"], ort.GraphOptimizationLevel.ORT_DISABLE_ALL)
for name, lv in levels:
w = run(["WebGpuExecutionProvider", "CPUExecutionProvider"], lv)
nf = 100.0 * (~np.isfinite(w)).mean()
d = np.nanmax(np.abs(np.where(np.isfinite(w), w, np.nan) - ref))
print(f"{name:<24}{nf:13.3f}%{d:20.3e}")
```
### Urgency
Not blocking — the `ORT_ENABLE_BASIC` workaround is sufficient for us. Filing it because the failure is silent: no error is raised, throughput is normal, and only a numerical comparison against another EP surfaces it.
### Platform
Mac
### OS Version
macOS 26.1 (Darwin 25.1.0), Apple M3 Pro
### ONNX Runtime Installation
Built from source
### ONNX Runtime Version or Commit ID
v1.29.0
### Execution Provider
Other / Unknown (WebGPU)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running decode.sh and repro.py from the linked gist to reproduce the non-finite WebGPU output across graph optimization levels. Then inspect the WebGPU EP's Conv+BatchNorm and Conv+Activation fusion path; done means the provided model produces finite results matching the CPU EP at EXTENDED and ALL without breaking the BASIC and disabled cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100