microsoft / microsoft/onnxruntime
WebGPU ConvTranspose produces incorrect results for fp16 models when output spatial dim > 2048 (index math done in f16)
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
**Describe the issue**
The WebGPU EP's `ConvTranspose` kernel produces incorrect output for **float16** models once an output spatial dimension exceeds ~2048. The error is *data-independent* and *size-dependent*: it grows with output size and the first incorrect output element appears around spatial position 2048. The same model run on the CPU EP (or on WebGPU in fp32) is correct.
This is **not** the same as the already-merged fix #27749 (non-vectorizable input channels); that code path is present in 1.25/1.26/main and does not cover this case.
**Root cause**
In `onnxruntime/core/providers/webgpu/nn/conv_backprop.cc`, `ConvTranspose2DProgram::GenerateShaderCode` computes the input-gradient indices `dyR`/`dyC` and their bounds checks using `dy_element_t`, which is the tensor's element type — **f16 for an fp16 model**:
```cpp
// lines ~108-110
"let dyR = (dy_element_t(dyRCorner) + dy_element_t(wR)) / dy_element_t(uniforms.strides[0]);\n"
...
"if (dyR < 0.0 || dyR >= dy_element_t(uniforms.dy_shape[" << row_dim << "]) || fract(dyR) > 0.0 || wRPerm < 0) {\n"
// lines ~123-126 (same pattern for the column axis)
"let dyC = (dy_element_t(dyCCorner) + dy_element_t(wC)) / dy_element_t(uniforms.strides.y);\n"
...
"if (dyC < 0.0 || dyC >= dy_element_t(uniforms.dy_shape[" << col_dim << "]) ||\n"
" fract(dyC) > 0.0 || wCPerm < 0) {\n"
```
f16 cannot represent consecutive integers exactly above 2048 (11-bit mantissa). For output positions beyond that range, `dy_element_t(dyRCorner)`, the division, the `fract()` divisibility test, and the `>= dy_shape` bounds check all operate on rounded values. This selects the wrong input element (or wrongly passes/fails the stride-divisibility and bounds checks), corrupting a band of outputs. Because it is integer-index arithmetic forced through f16, the defect depends only on **size**, not on the data.
**Reproduction**
- Any fp16 model containing a `ConvTranspose` whose output spatial dimension is > ~2048 (a 1-D ConvTranspose expressed as 2-D with H=1 reproduces it; e.g. in_channels=128, kernel=11, stride=5, input length ~1600 → output length ~8000).
- Run the node on the WebGPU EP vs the CPU EP on identical inputs and compare.
Observed relative error vs CPU as a function of output spatial size (single ConvTranspose, fp16):
| output spatial dim | rel. error |
|---|---|
| ≤ 996 | ~0% (correct) |
| 3996 | ~31% |
| 7996 | ~47% |
| 15996 | ~78% |
First incorrect element appears at output position ≈ 2048, consistent with the f16 integer-precision limit.
**Confirmed on:** ORT 1.25.0 and 1.26.0 (byte-identical divergence between the two), and the bug is still present on current `main` (`a1fc916579`). `git log v1.26.0..main` contains no WebGPU `conv_transpose` fix.
**Environment**
- EP: WebGPU (Dawn, Vulkan backend)
- GPU tested: Intel Iris Xe (Vulkan), Linux x64
- Affected dtype: float16 (fp32 models are unaffected)
**Suggested fix**
Compute `dyR`/`dyC` and their bounds checks in **f32** rather than `dy_element_t`. The index arithmetic is logically integer and should not be done in the storage element type. Four edits in `GenerateShaderCode` (the `dyR =`, `dyR >=`, `dyC =`, `dyC >=` expressions):
```cpp
let dyR = (f32(dyRCorner) + f32(wR)) / f32(uniforms.strides[0]);
...
if (dyR < 0.0 || dyR >= f32(uniforms.dy_shape[row_dim]) || fract(dyR) > 0.0 || wRPerm < 0) {
...
let dyC = (f32(dyCCorner) + f32(wC)) / f32(uniforms.strides.y);
...
if (dyC < 0.0 || dyC >= f32(uniforms.dy_shape[col_dim]) || fract(dyC) > 0.0 || wCPerm < 0) {
```
After this change (ORT 1.26, Intel Vulkan WebGPU): the size sweep above drops to a uniform ~0.31% across all sizes, and an isolated real-data ConvTranspose that was ~23% off drops to ~0.28%. The residual is ordinary fp16 drift, not a kernel defect.
Contributor guide
Research direction
Start in onnxruntime/core/providers/webgpu/nn/conv_backprop.cc at ConvTranspose2DProgram::GenerateShaderCode, focusing on the dyR and dyC index calculations and bounds checks. Reproduce the fp16 ConvTranspose case with an output spatial dimension above 2048 and compare WebGPU against the CPU EP. Done means large fp16 outputs agree with CPU while existing fp32 behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100