microsoft / microsoft/onnxruntime
Assertion failure in graph optimizer Level2 after MatmulTransposeFusion on large MoE model
@yuslepukhin is already working on this.
Since May 27, 2026.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
## Repro
Loading a Qwen3.5-MoE (40-layer, 256-expert, MoE with shared expert) bf16 ONNX exported with mobius using `--ep cuda --dtype f16` (or bf16) aborts during `InferenceSession` initialization right after `MatmulTransposeFusion` finishes.
## Symptom
```
[I:onnxruntime:, graph_transformer.cc:15 Apply] GraphTransformer MatmulTransposeFusion modified: 0 with status: OK
/opt/rh/gcc-toolset-14/root/usr/include/c++/14/bits/stl_vector.h:1130:
constexpr std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type)
[... NodeArg* ...]: Assertion '__n < this->size()' failed.
Aborted (core dumped)
```
## Env
- onnxruntime-gpu 1.27.0.dev20260511001 (CUDA 12)
- CUDAExecutionProvider on NVIDIA H200
- Model: Qwen/Qwen3.6-35B-A3B exported via mobius with `com.microsoft::GroupQueryAttention`, `com.microsoft::LinearAttention`, `com.microsoft::CausalConvWithState`
- 40-layer hybrid (full + linear attention) + 256-expert MoE per layer + shared expert
- Decoder model: ~64 GB (f16 with weights, 1.6 M nodes)
## What I tried
- `disabled_optimizers=['BiasGeluFusion','BiasSoftmaxFusion','BiasDropoutFusion','MatMulScaleFusion','SkipLayerNormFusion','LayerNormFusionL2','QuickGeluFusion','GeluApproximation']` — still aborts after MatmulTransposeFusion.
- `graph_optimization_level=ORT_ENABLE_BASIC` avoids the abort but then fails EP placement on `CausalConvWithState` (needs the function-inlining transformer that only runs at higher levels).
## Likely cause
Some Level2 GraphTransformer running between `MatmulTransposeFusion` and the next logged transformer reads past the end of a node-arg list — probably triggered by the unusual graph shape (10240+ MatMul nodes from per-expert MLPs, hybrid attention).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.