[None][feat] Ensure pre-fusion quantized graph uses SM-agnostic fake-quant custom ops

Summary

The AutoDeploy transform pipeline should guarantee that the pre-fusion graph — before any fusion transforms run — only contains fake-quant custom ops whose eager implementations are fully SM-agnostic (i.e., run correctly on any CUDA GPU regardless of compute capability). SM-specific optimized kernels should only be introduced by the fusion transforms that replace these fake-quant ops.

Related

• #12303 — Add GPU SM capability checks to all AutoDeploy quantization and MOE transforms (the fusion-side of the same problem)

Background & Motivation

AutoDeploy's quantization pipeline has two layers of custom ops:

Layer	Op naming convention	Purpose	SM requirement
Pre-fusion (fake quant)	torch_fake_quant_*	Simulate quantization effects (quant → dequant → float GEMM) for graph tracing, calibration, and correctness validation	Should be none
Post-fusion (real quant)	torch_quant_* / trtllm_quant_*	Optimized fused kernels for inference	SM-specific (e.g., FP8 ≥ 89, NVFP4 ≥ 100)

Today, the fake-quant ops (e.g., torch_fake_quant_nvfp4_linear, torch_fake_quant_fp8_linear) happen to use pure PyTorch math and are SM-agnostic. However, this is not enforced or documented as an invariant, and it would be easy to accidentally introduce an SM-specific dependency (e.g., a triton kernel, a TRT-LLM op call) into a fake-quant implementation.

Making this a hard architectural requirement gives us:

1. Testability on any GPU: Pre-fusion graph correctness can be validated on Hopper even for NVFP4 models, or on Ampere for FP8 models, which greatly expands CI coverage.
2. Debuggability: Users can run --compile-backend=torch-simple (no fusion) on any available GPU to isolate whether a bug is in quantization logic vs. fused kernels.
3. Clean separation of concerns: Quantization numerics (how values are scaled/rounded/packed) are decoupled from hardware-specific kernel dispatch.

Current State

Fake-quant ops (pre-fusion)

Op	Implementation	SM-agnostic today?
torch_fake_quant_fp8_linear	Pure PyTorch (dequant + matmul)	Yes
torch_fake_quant_nvfp4_linear	Pure PyTorch (_quantize_nvfp4 / _dequantize_nvfp4 + matmul)	Yes
torch_fake_quant_int4_linear	Pure PyTorch (unpack + dequant + matmul)	Yes
torch_fake_quant_int4_gptq_linear	Pure PyTorch (unpack + dequant + matmul)	Yes
torch_fake_quant_finegrained_fp8_linear	Pure PyTorch (block-wise dequant + matmul)	Yes

Real/fused ops (post-fusion) — introduced by fusion transforms

Op	SM dependency
torch_quant_fp8_linear	addmm_float8_unwrapped (SM ≥ 89)
trtllm_quant_fp8_linear	TRT-LLM scaled_mm (SM ≥ 89)
torch_quant_nvfp4_linear	torch.ops.trtllm.fp4_quantize + nvfp4_gemm (SM ≥ 100)
trtllm_finegrained_fp8_linear	fp8_block_scaling_gemm (SM ≥ 90)

MOE ops (similar pattern needed)

The same fake-quant → real-quant layering should apply to MOE ops (torch_quant_fp8_moe, torch_quant_nvfp4_moe, etc.), but the MOE path currently may not have the same clean separation.

Proposed Changes

1. Document the invariant

Add a clear docstring/comment in custom_ops/quantization/torch_quant.py (where fake-quant ops live) stating that all torch_fake_quant_* ops must use only SM-agnostic PyTorch operations. No triton kernels, no TRT-LLM ops, no torch.ops.trtllm.* calls.

2. Ensure all quant formats have SM-agnostic fake-quant ops

Verify that every quantization format supported by AutoDeploy has a corresponding torch_fake_quant_* op that can run on any GPU:

• FP8 (standard per-tensor)
• NVFP4
• INT4 / INT4-GPTQ
• FineGrained FP8 (block-scaled)
• MXFP4 — check if a fake-quant path exists
• Any future formats

3. Extend pattern to MOE

Ensure MOE quantization ops follow the same two-layer pattern:

• Pre-fusion: torch_fake_quant_*_moe ops using generic PyTorch (SM-agnostic)
• Post-fusion: torch_quant_*_moe / trtllm_quant_*_moe ops using optimized kernels (SM-specific)

4. Add CI validation

Consider adding a test that runs a quantized model config through the pre-fusion graph (without fusion transforms) on a lower-SM GPU to validate that the fake-quant path doesn't accidentally depend on high-SM features.

Acceptance Criteria

• All torch_fake_quant_* op implementations use only SM-agnostic PyTorch operations
• Docstring invariant added to torch_quant.py or a shared location documenting the rule
• MOE fake-quant ops follow the same SM-agnostic pattern
• Pre-fusion graph can execute on any CUDA GPU (e.g., Ampere/Hopper) for any quant format, including NVFP4
• CI test validating the above (optional but recommended)