microsoft / microsoft/onnxruntime

WebGPU ConvTranspose produces incorrect results for fp16 models when output spatial dim > 2048 (index math done in f16)

Open Beginner friendly
#28,976 3 comments 0 reactions 0 assignees View on GitHub
ep:WebGPU platform:web stale
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

**Describe the issue**

The WebGPU EP's `ConvTranspose` kernel produces incorrect output for **float16** models once an output spatial dimension exceeds ~2048. The error is *data-independent* and *size-dependent*: it grows with output size and the first incorrect output element appears around spatial position 2048. The same model run on the CPU EP (or on WebGPU in fp32) is correct.

This is **not** the same as the already-merged fix #27749 (non-vectorizable input channels); that code path is present in 1.25/1.26/main and does not cover this case.

**Root cause**

In `onnxruntime/core/providers/webgpu/nn/conv_backprop.cc`, `ConvTranspose2DProgram::GenerateShaderCode` computes the input-gradient indices `dyR`/`dyC` and their bounds checks using `dy_element_t`, which is the tensor's element type — **f16 for an fp16 model**:

```cpp
// lines ~108-110
"let dyR = (dy_element_t(dyRCorner) + dy_element_t(wR)) / dy_element_t(uniforms.strides[0]);\n"
...
"if (dyR < 0.0 || dyR >= dy_element_t(uniforms.dy_shape[" << row_dim << "]) || fract(dyR) > 0.0 || wRPerm < 0) {\n"

// lines ~123-126 (same pattern for the column axis)
"let dyC = (dy_element_t(dyCCorner) + dy_element_t(wC)) / dy_element_t(uniforms.strides.y);\n"
...
"if (dyC < 0.0 || dyC >= dy_element_t(uniforms.dy_shape[" << col_dim << "]) ||\n"
" fract(dyC) > 0.0 || wCPerm < 0) {\n"
```

f16 cannot represent consecutive integers exactly above 2048 (11-bit mantissa). For output positions beyond that range, `dy_element_t(dyRCorner)`, the division, the `fract()` divisibility test, and the `>= dy_shape` bounds check all operate on rounded values. This selects the wrong input element (or wrongly passes/fails the stride-divisibility and bounds checks), corrupting a band of outputs. Because it is integer-index arithmetic forced through f16, the defect depends only on **size**, not on the data.

**Reproduction**

- Any fp16 model containing a `ConvTranspose` whose output spatial dimension is > ~2048 (a 1-D ConvTranspose expressed as 2-D with H=1 reproduces it; e.g. in_channels=128, kernel=11, stride=5, input length ~1600 → output length ~8000).
- Run the node on the WebGPU EP vs the CPU EP on identical inputs and compare.

Observed relative error vs CPU as a function of output spatial size (single ConvTranspose, fp16):

| output spatial dim | rel. error |
|---|---|
| ≤ 996 | ~0% (correct) |
| 3996 | ~31% |
| 7996 | ~47% |
| 15996 | ~78% |

First incorrect element appears at output position ≈ 2048, consistent with the f16 integer-precision limit.

**Confirmed on:** ORT 1.25.0 and 1.26.0 (byte-identical divergence between the two), and the bug is still present on current `main` (`a1fc916579`). `git log v1.26.0..main` contains no WebGPU `conv_transpose` fix.

**Environment**
- EP: WebGPU (Dawn, Vulkan backend)
- GPU tested: Intel Iris Xe (Vulkan), Linux x64
- Affected dtype: float16 (fp32 models are unaffected)

**Suggested fix**

Compute `dyR`/`dyC` and their bounds checks in **f32** rather than `dy_element_t`. The index arithmetic is logically integer and should not be done in the storage element type. Four edits in `GenerateShaderCode` (the `dyR =`, `dyR >=`, `dyC =`, `dyC >=` expressions):

```cpp
let dyR = (f32(dyRCorner) + f32(wR)) / f32(uniforms.strides[0]);
...
if (dyR < 0.0 || dyR >= f32(uniforms.dy_shape[row_dim]) || fract(dyR) > 0.0 || wRPerm < 0) {
...
let dyC = (f32(dyCCorner) + f32(wC)) / f32(uniforms.strides.y);
...
if (dyC < 0.0 || dyC >= f32(uniforms.dy_shape[col_dim]) || fract(dyC) > 0.0 || wCPerm < 0) {
```

After this change (ORT 1.26, Intel Vulkan WebGPU): the size sweep above drops to a uniform ~0.31% across all sizes, and an isolated real-data ConvTranspose that was ~23% off drops to ~0.28%. The residual is ordinary fp16 drift, not a kernel defect.

Contributor guide

Open the contributing guide

Research direction

Start in onnxruntime/core/providers/webgpu/nn/conv_backprop.cc at ConvTranspose2DProgram::GenerateShaderCode, focusing on the dyR and dyC index calculations and bounds checks. Reproduce the fp16 ConvTranspose case with an output spatial dimension above 2048 and compare WebGPU against the CPU EP. Done means large fp16 outputs agree with CPU while existing fp32 behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.