microsoft / microsoft/onnxruntime

DirectML EP miscomputes the Split -> Sigmoid -> Mul pattern emitted by nn.GLU

Open
#32,146 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ep:DML stale
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

## Summary

The DirectML execution provider silently miscomputes the two-way `Split -> Sigmoid -> Mul`
pattern emitted when `torch.nn.GLU` is exported to ONNX. The CPU execution provider matches
PyTorch and NumPy. On DirectML, one split output is reused where the other is required; in the
characterized RoFormer graph this behaved as `a * sigmoid(a)` instead of `a * sigmoid(b)`. No
error or fallback is reported.

The diagnostic fingerprint is a minimum output value of approximately `-0.2785`, the minimum of
`x * sigmoid(x)`. Disabling ONNX Runtime graph optimizations does not change the result.

## Environment

- Windows 11 Pro 25H2, build 26200.9168
- AMD Radeon RX 7800 XT
- Python 3.11
- `onnxruntime-directml==1.24.4`
- `torch==2.13.0+cpu`
- `onnx==1.22.0`
- `numpy==2.4.6`

## Minimal reproduction

```python
import io
import numpy as np
import onnxruntime as ort
import torch

n = 4096
a = np.zeros((1, n), dtype=np.float32)
b = np.linspace(-4, 4, n, dtype=np.float32)[None]
x = np.concatenate((a, b), axis=-1)
model = io.BytesIO()
torch.onnx.export(torch.nn.GLU(-1), torch.from_numpy(x), model,
opset_version=17, dynamo=False)
opts = ort.SessionOptions()
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
expected = a * (1 / (1 + np.exp(-b)))
for ep in ("CPUExecutionProvider", "DmlExecutionProvider"):
session = ort.InferenceSession(model.getvalue(), opts, providers=[ep])
actual = session.run(None, {session.get_inputs()[0].name: x})[0]
print(ep, "max_err=", np.max(np.abs(actual - expected)), "min=", actual.min())
```

Output:

```text
CPUExecutionProvider max_err= 0.0 min= 0.0
DmlExecutionProvider max_err= 3.928055 min= -0.27846456
```

## Expected behavior

`DmlExecutionProvider` should match the CPU execution provider and compute the GLU output from
the first half multiplied by the sigmoid of the second half. In the reproduction above, the
first half (`a`) is zero, so every output value should be zero.

## Actual behavior

DirectML returns non-zero values with a minimum of `-0.27846456`. This is the characteristic
minimum of a self-gated tensor, `x * sigmoid(x)`. With the input ordering in this minimal export,
the output behaves as `b * sigmoid(b)` instead of `a * sigmoid(b)`, demonstrating that the split
output feeding `Sigmoid` is also reused by `Mul`. In the original RoFormer graph, the equivalent
aliasing manifested as `a * sigmoid(a)` instead of `a * sigmoid(b)`.

## Characterization

The behavior was isolated with small ONNX graphs:

| Pattern | CPU EP | DirectML EP |
| --- | --- | --- |
| Two-way `Split`, `Sigmoid` on one output, then `Mul` with the other | Correct | Incorrect |
| Same graph with a positive axis instead of `-1` | Correct | Incorrect |
| Split operands exchanged | Correct | Correct |
| `Add` instead of `Mul` | Correct | Correct |
| Four-way split with two GLUs summed | Correct | Correct |
| 62-way split and concat used by the band-split block | Correct | Correct |
| Two `Slice` operations instead of the two-way `Split` | Correct | Correct |

The incorrect result is unchanged with
`SessionOptions.graph_optimization_level = ORT_DISABLE_ALL`, so the issue does not appear to be
caused by an ONNX Runtime graph rewrite or fusion.

## Workaround and model-level impact

The port replaces each `nn.GLU` during export with two `Slice` operations followed by `Sigmoid`
and `Mul` (`toolkit/spec_models.py`). The release `manifest.json` records `glu_replaced=60`.
With that workaround, DirectML parity against the unmodified PyTorch model improved from a maximum
absolute error of `5.76` to `6.7e-06`.

This can silently affect other ONNX models exported from `nn.GLU`, an operator pattern used in
audio and NLP architectures. The observed scope is the environment above; other DirectML hardware
and driver combinations have not yet been characterized.

---

Found while porting BS-RoFormer / Mel-Band RoFormer to ONNX: https://github.com/santiquiroz/port-bs-roformer-onnx — the full characterization and the workaround live there if more context helps.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the minimal Python reproduction with CPUExecutionProvider and DmlExecutionProvider, then trace DirectML execution of the two-way Split -> Sigmoid -> Mul pattern. The related workaround is in toolkit/spec_models.py; done means DirectML matches the CPU result for the GLU reproduction without requiring Slice-based replacement.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.