microsoft / microsoft/onnxruntime
[Bug] GemmTransposeFusion folds an identity Transpose (perm=[0,1]) into Gemm transA/transB, producing silently wrong results
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
`Transpose` with `perm=[0,1]` on a rank-2 tensor is the identity. When such a
node feeds a `MatMul` that is followed by `Add`, ORT fuses the pattern into a
single `Gemm` and sets `transA=1` (or `transB=1`), so the fused node computes
`A^T @ B + C` instead of `A @ B + C`.
Consequences depend only on shapes:
* `M == K` (square operand): the model **silently returns wrong numbers**.
* `M != K`: execution fails with
`InvalidArgument ... Gemm: Invalid bias shape for broadcast`,
reported on a node named `/MatMulAddFusion/GemmTransposeFusion/`.
With `ORT_DISABLE_ALL` the same model always produces the correct result, so
this is purely a graph-optimization defect.
This is a **regression**: onnxruntime 1.20.1 is correct, 1.27.0 is not.
### Root cause
`onnxruntime/core/optimizer/gemm_transpose_fusion.cc` decides to fold based on
the operator type alone and never inspects the `perm` attribute:
```cpp
if (A_node_ptr != nullptr && A_node_ptr->OpType() == "Transpose") {
...
transA = !transA;
}
```
Any `Transpose` toggles the flag, including a `perm` that is the identity
permutation. The fusion should only fire when `perm == [1, 0]` for rank-2
inputs (and should treat identity `perm` as a no-op that can simply be removed).
Dumping the optimized graph confirms the wrong attribute:
| input graph | optimized graph | correct? |
|---|---|---|
| `Transpose(perm=[0,1])` → MatMul → Add | `Gemm{transA:1}` | **no** |
| `Transpose(perm=[1,0])` → MatMul → Add | `Gemm{transA:1}` | yes |
| MatMul → Add | `Gemm{transA:0}` | yes |
| `Transpose(perm=[0,1])` → MatMul (no Add) | `FusedMatMul{transA:0}` | yes |
The no-`Add` path (`MatMulTransposeFusion` → `FusedMatMul`) handles the identity
`perm` correctly, so only the `Gemm` path is affected.
### Expected behavior
Both optimization levels return `x @ y + b`.
### Scope confirmed
* Wrong on the `A` side and on the `B` side of the `MatMul`.
* Wrong for `float32` and `float16`; `float64` unaffected.
* Deterministic: 5/5 identical repeats.
* `CPUExecutionProvider` and `CoreMLExecutionProvider` both affected.
* Shapes checked: `(3,3,3) (4,4,4) (5,5,5) (4,4,2) (8,8,3)` silent wrong result;
`(2,3,2) (3,4,5)` execution error.
### To reproduce
```python
import numpy as np, onnx, onnxruntime as ort
from onnx import helper, TensorProto
F = TensorProto.FLOAT
M = K = N = 3
nodes = [helper.make_node("Transpose", ["x"], ["t"], perm=[0, 1]), # identity
helper.make_node("MatMul", ["t", "y"], ["mm"]),
helper.make_node("Add", ["mm", "b"], ["o"])]
g = helper.make_graph(nodes, "g",
[helper.make_tensor_value_info("x", F, [M, K]),
helper.make_tensor_value_info("y", F, [K, N]),
helper.make_tensor_value_info("b", F, [M, N])],
[helper.make_tensor_value_info("o", F, [M, N])])
m = helper.make_model(g, opset_imports=[helper.make_opsetid("", 21)], ir_version=10)
onnx.checker.check_model(m, full_check=True)
rng = np.random.default_rng(0)
feed = {"x": rng.standard_normal((M, K)).astype("float32"),
"y": rng.standard_normal((K, N)).astype("float32"),
"b": rng.standard_normal((M, N)).astype("float32")}
expected = feed["x"] @ feed["y"] + feed["b"]
for lvl in (ort.GraphOptimizationLevel.ORT_DISABLE_ALL,
ort.GraphOptimizationLevel.ORT_ENABLE_ALL):
so = ort.SessionOptions(); so.graph_optimization_level = lvl
got = ort.InferenceSession(m.SerializeToString(), so,
providers=["CPUExecutionProvider"]).run(None, feed)[0]
print(lvl, "match =", np.allclose(got, expected, atol=1e-5))
```
Output on 1.27.0:
```text
GraphOptimizationLevel.ORT_DISABLE_ALL match = True
GraphOptimizationLevel.ORT_ENABLE_ALL match = False
```
Using `M = 2, K = 3, N = 2` turns the silent mismatch into
`InvalidArgument ... Gemm: Invalid bias shape for broadcast`.
### Urgency
Medium-high. Identity transposes are routinely emitted by ONNX exporters and by
graph-rewriting tools, and the square-shape case corrupts inference results with
no error reported.
### Platform
Mac
### OS Version
macOS 26.0
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.27.0 (broken); 1.20.1 (correct)
### ONNX Runtime API
Python
### Architecture
ARM64
### Execution Provider
Default CPU, CoreML
### Execution Provider Library Version
N/A
Contributor guide
Research direction
Start in onnxruntime/core/optimizer/gemm_transpose_fusion.cc and trace how the Transpose node's perm attribute affects transA and transB. Reproduce the issue with the provided Python model at ORT_ENABLE_ALL, then verify that identity transposes no longer produce an incorrect Gemm result while perm=[1,0] still fuses correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100