microsoft / microsoft/onnxruntime
[Feature Request] Add Fusion Transformers for WebNN EP Decomposed LM Operators
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
We propose adding fusion transformers to reconstruct LM operators decomposed during WebNN EP graph processing: GroupQueryAttention, MatmulNBits, RMSNorm and GatherBlockQuantized.
These high-level operators are already present in the original ONNX model, but the WebNN EP decomposes them into primitive operations to meet WebNN API constraints. This decomposition affects both performance and numerical accuracy: native EPs such as WebGPU can no longer take advantage of their specialized kernels for these high-level operators.
### Describe scenario use case
The target scenario is on-device LM inference through WebNN, using WebNN's ORT backend with native WebGPU EP. In phi4-mini demo, reconstructing these operators reduces graph complexity:
| Operator | Node reduction |
|---|---|
| GroupQueryAttention | About 3,030 GQA-related primitive nodes → 32 GQA nodes |
| MatMulNBits | 644 → 161 nodes |
| RMSNorm | 390 → 65 nodes |
| GatherBlockQuantized | 2 → 1 node, avoiding a roughly 1.145 GiB dequantized embedding table per invocation |
And these fusions improve decode IPOT from ~2000ms/token to ~60ms/token. The upstream path also exhibited accuracy issues; restoring RMSNorm with FP32 statistics resolved the generation correctness issue observed in our demo.
Contributor guide
Research direction
Start by tracing how the WebNN EP decomposes GroupQueryAttention, MatmulNBits, RMSNorm, and GatherBlockQuantized, then compare that path with the native WebGPU EP. Use the phi4-mini demo as the validation scenario. Done means reconstructing the high-level operators, reducing graph complexity, improving decode latency, and preserving generation correctness with accurate RMSNorm statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100