microsoft / microsoft/onnxruntime

[Feature Request] Add Fusion Transformers for WebNN EP Decomposed LM Operators

Open
#32,496 4 comments 2 reactions 0 assignees View on GitHub
ep:WebGPU ep:WebNN feature request model:transformer platform:web
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

We propose adding fusion transformers to reconstruct LM operators decomposed during WebNN EP graph processing: GroupQueryAttention, MatmulNBits, RMSNorm and GatherBlockQuantized.

These high-level operators are already present in the original ONNX model, but the WebNN EP decomposes them into primitive operations to meet WebNN API constraints. This decomposition affects both performance and numerical accuracy: native EPs such as WebGPU can no longer take advantage of their specialized kernels for these high-level operators.

### Describe scenario use case

The target scenario is on-device LM inference through WebNN, using WebNN's ORT backend with native WebGPU EP. In phi4-mini demo, reconstructing these operators reduces graph complexity:

| Operator | Node reduction |
|---|---|
| GroupQueryAttention | About 3,030 GQA-related primitive nodes → 32 GQA nodes |
| MatMulNBits | 644 → 161 nodes |
| RMSNorm | 390 → 65 nodes |
| GatherBlockQuantized | 2 → 1 node, avoiding a roughly 1.145 GiB dequantized embedding table per invocation |

And these fusions improve decode IPOT from ~2000ms/token to ~60ms/token. The upstream path also exhibited accuracy issues; restoring RMSNorm with FP32 statistics resolved the generation correctness issue observed in our demo.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the WebNN EP decomposes GroupQueryAttention, MatmulNBits, RMSNorm, and GatherBlockQuantized, then compare that path with the native WebGPU EP. Use the phi4-mini demo as the validation scenario. Done means reconstructing the high-level operators, reducing graph complexity, improving decode latency, and preserving generation correctness with accurate RMSNorm statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.