[Feature Request] Improved pattern matcher for AR+residual_add+RMSNorm fusion (cast/reshape coverage)
@greg-kwasniewski1 is already working on this.
Since May 30, 2026.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Summary
The AutoDeploy fuse_allreduce_residual_rmsnorm transform only matches the exact chain all_reduce -> add(residual) -> rmsnorm. On real FP8 / MoE models the exported graph interposes either a shape-only reshape/view (between the all-reduce and the residual add) or a dtype aten.to.dtype cast (between the residual add and the RMSNorm). The matcher then silently misses the site and the production fused AR kernel is never used.
Measured impact
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8(TP=4): 52trtllm_dist_all_reducenodes, fusion matched 1/52 — the residual stream is carried throughadd -> aten.to.dtype(bf16) -> rmsnorm.Qwen/Qwen3.5-35B-A3B-FP8(TP=4): 80 all-reduce sites, matched 40/80 — the 40 MLP/MoE sites areall_reduce -> aten.reshape -> add -> rmsnorm.
This disables fusion for both the trtllm and the experimental cute backends, since they share this matcher.
Proposed approach
- Cast variant: add
all_reduce -> add -> aten.to.dtype -> rmsnormpattern(s), rewiring the fused op's residual (second) output onto the cast node. The fused kernels already emit a bf16 residual, so the cast is absorbed. The matcher util itself flagsaten.to.dtypenode elision (utils/pattern_matcher.py,register_ad_patterndocstring point 3c) — handle via a dedicated pattern +op_ignore_typeson the cast dtype arg. - Reshape variant: tolerate a shape-only
aten.reshape/viewbetween the all-reduce and the residual add; fold it into the fused op (ignore the shape literals viaop_ignore_types). - Keep existing variants; parameterize the factory over {add_order} x {rmsnorm_op} x {cast?} x {reshape?}.
Scope
- Surface:
tensorrt_llm/_torch/auto_deploy/transform/library/collectives.py - Tests:
tests/unittest/auto_deploy/.../test_allreduce_residual_rmsnorm_fusion.py
Tests / validation
- Graph-level match-count unit test (CPU/meta): synthetic modules with an intervening cast and reshape; assert the fused op appears and
num_matchesincreases. - Extend the existing multi-GPU correctness test with the cast and reshape cases.
Risk
library-visible (matcher shared by the trtllm + cute fusion backends; AutoDeploy multi-GPU CI stages required). Not API-visible.
Links
- Related PR: (filled in once the PR is opened)
- Discovered during evaluation of PR #14534 (experimental CuTe fused AR+residual+RMSNorm kernel).
- JIRA: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.