NVIDIA / NVIDIA/TensorRT-LLM

[Feature Request] Improved pattern matcher for AR+residual_add+RMSNorm fusion (cast/reshape coverage)

Open
#14,781 3 comments 0 reactions 1 assignee View on GitHub

@greg-kwasniewski1 is already working on this.

Since May 30, 2026.

AutoDeploy Customized kernels feature request
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Summary

The AutoDeploy fuse_allreduce_residual_rmsnorm transform only matches the exact chain all_reduce -> add(residual) -> rmsnorm. On real FP8 / MoE models the exported graph interposes either a shape-only reshape/view (between the all-reduce and the residual add) or a dtype aten.to.dtype cast (between the residual add and the RMSNorm). The matcher then silently misses the site and the production fused AR kernel is never used.

Measured impact

  • nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 (TP=4): 52 trtllm_dist_all_reduce nodes, fusion matched 1/52 — the residual stream is carried through add -> aten.to.dtype(bf16) -> rmsnorm.
  • Qwen/Qwen3.5-35B-A3B-FP8 (TP=4): 80 all-reduce sites, matched 40/80 — the 40 MLP/MoE sites are all_reduce -> aten.reshape -> add -> rmsnorm.

This disables fusion for both the trtllm and the experimental cute backends, since they share this matcher.

Proposed approach

  • Cast variant: add all_reduce -> add -> aten.to.dtype -> rmsnorm pattern(s), rewiring the fused op's residual (second) output onto the cast node. The fused kernels already emit a bf16 residual, so the cast is absorbed. The matcher util itself flags aten.to.dtype node elision (utils/pattern_matcher.py, register_ad_pattern docstring point 3c) — handle via a dedicated pattern + op_ignore_types on the cast dtype arg.
  • Reshape variant: tolerate a shape-only aten.reshape/view between the all-reduce and the residual add; fold it into the fused op (ignore the shape literals via op_ignore_types).
  • Keep existing variants; parameterize the factory over {add_order} x {rmsnorm_op} x {cast?} x {reshape?}.

Scope

  • Surface: tensorrt_llm/_torch/auto_deploy/transform/library/collectives.py
  • Tests: tests/unittest/auto_deploy/.../test_allreduce_residual_rmsnorm_fusion.py

Tests / validation

  • Graph-level match-count unit test (CPU/meta): synthetic modules with an intervening cast and reshape; assert the fused op appears and num_matches increases.
  • Extend the existing multi-GPU correctness test with the cast and reshape cases.

Risk

library-visible (matcher shared by the trtllm + cute fusion backends; AutoDeploy multi-GPU CI stages required). Not API-visible.

Links

  • Related PR: (filled in once the PR is opened)
  • Discovered during evaluation of PR #14534 (experimental CuTe fused AR+residual+RMSNorm kernel).
  • JIRA: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.