[AutoDeploy] Standardized MoE handling + onboarding
Open
Nobody has claimed this yet.
AutoDeploy
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
MoE Interface discussion
- 3 stages as shown below
- Each stage should be one or a set of self-contained transforms
- Separate each stage into separate file in the transform library
Stage 1 (mapping to unquantized reference op)
# THIS STAGE: EVERYTHING GOES TO torch_moe op
match_moe_pattern:
# TODO: deprecate this one since not used anymore
stage: pattern_matcher
match_dense_moe_pattern:
# TODO: make this one match to torch_moe instead of torch_moe_dense_mlp (used by GPT-OSS)
stage: pattern_matcher
match_bmm_moe_pattern:
# TODO: put functionality into match_dense_moe_pattern and deprecate this one (used by Llama4)
# TODO: understand if load hook functionality is needed and/or can be integrated into match_dense_moe_pattern
stage: pattern_matcher
Stage 2 (mapping to quantized reference op)
# THIS STAGE: GOES FROM torch_moe TO torch_quant_*_moe op (torch_quant_fp8_moe, torch_quant_nvfp4_moe, torch_quant_mxfp4_moe)
quantize_fp8_moe:
stage: pattern_matcher
quantize_nvfp4_moe:
stage: pattern_matcher
quantize_mxfp4_moe:
stage: pattern_matcher
Stage 3 (mapping to optimal, fused kernels)
# THIS STAGE: map to optimal kernel backend for MoE (triton, trtllm, cutlass, etc...). from torch_**_moe to {backend}_**_moe
fuse_moe:
stage: post_load_fusion
enabled: true
backend: trtllm
fuse_fp8_moe:
stage: post_load_fusion
enabled: true
backend: trtllm
fuse_nvfp4_moe:
stage: post_load_fusion
enabled: false
# TODO: Implement this one, doesn't exist yet
fuse_mxfp4_moe:
stage: post_load_fusion
enabled: false
backend: triton
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.