microsoft / microsoft/onnxruntime

[Bug] GemmTransposeFusion folds an identity Transpose (perm=[0,1]) into Gemm transA/transB, producing silently wrong results

Open
#32,228 0 comments 0 reactions 0 assignees View on GitHub
ep:CoreML
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

`Transpose` with `perm=[0,1]` on a rank-2 tensor is the identity. When such a
node feeds a `MatMul` that is followed by `Add`, ORT fuses the pattern into a
single `Gemm` and sets `transA=1` (or `transB=1`), so the fused node computes
`A^T @ B + C` instead of `A @ B + C`.

Consequences depend only on shapes:

* `M == K` (square operand): the model **silently returns wrong numbers**.
* `M != K`: execution fails with
`InvalidArgument ... Gemm: Invalid bias shape for broadcast`,
reported on a node named `/MatMulAddFusion/GemmTransposeFusion/`.

With `ORT_DISABLE_ALL` the same model always produces the correct result, so
this is purely a graph-optimization defect.

This is a **regression**: onnxruntime 1.20.1 is correct, 1.27.0 is not.

### Root cause

`onnxruntime/core/optimizer/gemm_transpose_fusion.cc` decides to fold based on
the operator type alone and never inspects the `perm` attribute:

```cpp
if (A_node_ptr != nullptr && A_node_ptr->OpType() == "Transpose") {
...
transA = !transA;
}
```

Any `Transpose` toggles the flag, including a `perm` that is the identity
permutation. The fusion should only fire when `perm == [1, 0]` for rank-2
inputs (and should treat identity `perm` as a no-op that can simply be removed).

Dumping the optimized graph confirms the wrong attribute:

| input graph | optimized graph | correct? |
|---|---|---|
| `Transpose(perm=[0,1])` → MatMul → Add | `Gemm{transA:1}` | **no** |
| `Transpose(perm=[1,0])` → MatMul → Add | `Gemm{transA:1}` | yes |
| MatMul → Add | `Gemm{transA:0}` | yes |
| `Transpose(perm=[0,1])` → MatMul (no Add) | `FusedMatMul{transA:0}` | yes |

The no-`Add` path (`MatMulTransposeFusion` → `FusedMatMul`) handles the identity
`perm` correctly, so only the `Gemm` path is affected.

### Expected behavior

Both optimization levels return `x @ y + b`.

### Scope confirmed

* Wrong on the `A` side and on the `B` side of the `MatMul`.
* Wrong for `float32` and `float16`; `float64` unaffected.
* Deterministic: 5/5 identical repeats.
* `CPUExecutionProvider` and `CoreMLExecutionProvider` both affected.
* Shapes checked: `(3,3,3) (4,4,4) (5,5,5) (4,4,2) (8,8,3)` silent wrong result;
`(2,3,2) (3,4,5)` execution error.

### To reproduce

```python
import numpy as np, onnx, onnxruntime as ort
from onnx import helper, TensorProto

F = TensorProto.FLOAT
M = K = N = 3
nodes = [helper.make_node("Transpose", ["x"], ["t"], perm=[0, 1]), # identity
helper.make_node("MatMul", ["t", "y"], ["mm"]),
helper.make_node("Add", ["mm", "b"], ["o"])]
g = helper.make_graph(nodes, "g",
[helper.make_tensor_value_info("x", F, [M, K]),
helper.make_tensor_value_info("y", F, [K, N]),
helper.make_tensor_value_info("b", F, [M, N])],
[helper.make_tensor_value_info("o", F, [M, N])])
m = helper.make_model(g, opset_imports=[helper.make_opsetid("", 21)], ir_version=10)
onnx.checker.check_model(m, full_check=True)

rng = np.random.default_rng(0)
feed = {"x": rng.standard_normal((M, K)).astype("float32"),
"y": rng.standard_normal((K, N)).astype("float32"),
"b": rng.standard_normal((M, N)).astype("float32")}
expected = feed["x"] @ feed["y"] + feed["b"]

for lvl in (ort.GraphOptimizationLevel.ORT_DISABLE_ALL,
ort.GraphOptimizationLevel.ORT_ENABLE_ALL):
so = ort.SessionOptions(); so.graph_optimization_level = lvl
got = ort.InferenceSession(m.SerializeToString(), so,
providers=["CPUExecutionProvider"]).run(None, feed)[0]
print(lvl, "match =", np.allclose(got, expected, atol=1e-5))
```

Output on 1.27.0:

```text
GraphOptimizationLevel.ORT_DISABLE_ALL match = True
GraphOptimizationLevel.ORT_ENABLE_ALL match = False
```

Using `M = 2, K = 3, N = 2` turns the silent mismatch into
`InvalidArgument ... Gemm: Invalid bias shape for broadcast`.

### Urgency

Medium-high. Identity transposes are routinely emitted by ONNX exporters and by
graph-rewriting tools, and the square-shape case corrupts inference results with
no error reported.

### Platform

Mac

### OS Version

macOS 26.0

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.27.0 (broken); 1.20.1 (correct)

### ONNX Runtime API

Python

### Architecture

ARM64

### Execution Provider

Default CPU, CoreML

### Execution Provider Library Version

N/A

Contributor guide

Open the contributing guide

Research direction

Start in onnxruntime/core/optimizer/gemm_transpose_fusion.cc and trace how the Transpose node's perm attribute affects transA and transB. Reproduce the issue with the provided Python model at ORT_ENABLE_ALL, then verify that identity transposes no longer produce an incorrect Gemm result while perm=[1,0] still fuses correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.