NVIDIA / NVIDIA/TensorRT-LLM

[AutoDeploy] Standardized MoE handling + onboarding

Open
#4,333 2 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

AutoDeploy
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

MoE Interface discussion
  • 3 stages as shown below
  • Each stage should be one or a set of self-contained transforms
  • Separate each stage into separate file in the transform library
Stage 1 (mapping to unquantized reference op)
  # THIS STAGE: EVERYTHING GOES TO torch_moe op
  match_moe_pattern:
    # TODO: deprecate this one since not used anymore
    stage: pattern_matcher
  match_dense_moe_pattern:
    # TODO: make this one match to torch_moe instead of torch_moe_dense_mlp (used by GPT-OSS)
    stage: pattern_matcher
  match_bmm_moe_pattern:
    # TODO: put functionality into match_dense_moe_pattern and deprecate this one (used by Llama4)
    # TODO: understand if load hook functionality is needed and/or can be integrated into match_dense_moe_pattern
    stage: pattern_matcher
Stage 2 (mapping to quantized reference op)
# THIS STAGE: GOES FROM torch_moe TO torch_quant_*_moe op (torch_quant_fp8_moe, torch_quant_nvfp4_moe, torch_quant_mxfp4_moe)
  quantize_fp8_moe:
    stage: pattern_matcher
  quantize_nvfp4_moe:
    stage: pattern_matcher
  quantize_mxfp4_moe:
    stage: pattern_matcher
Stage 3 (mapping to optimal, fused kernels)
  # THIS STAGE: map to optimal kernel backend for MoE (triton, trtllm, cutlass, etc...). from torch_**_moe to {backend}_**_moe
  fuse_moe:
    stage: post_load_fusion
    enabled: true
    backend: trtllm
  fuse_fp8_moe:
    stage: post_load_fusion
    enabled: true
    backend: trtllm
  fuse_nvfp4_moe:
    stage: post_load_fusion
    enabled: false
  # TODO: Implement this one, doesn't exist yet
  fuse_mxfp4_moe:
    stage: post_load_fusion
    enabled: false
    backend: triton

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.