[Usage]: Support for MXFP4 W4A8 quantization on Hopper

System Info

Nvidia H100 GPUs, Intel Xeon 8480C CPU, NGC 26.01 PyTorch image (CUDA 13.1.1, PyTorch 2.10.0a0), TensorRT-LLM 1.3.0rc8.

How would you like to use TensorRT-LLM

I wanted to inquire regarding support for MXFP4 W4A8 quantization in TensorRT-LLM (as of 1.3.0rc8), as it is unclear whether this is supported or not for either dense or MoE models.

I made a few tests in this regard. Here you can find a ModelOpt-based minimal script which produces an MXFP4 or NVFP4 W4A8 quantized checkpoint, with a variety of options. It can be invoked as follows:

python hf_mxfp4_ptq.py --model <HF_INPUT_DIR> --quantized-ckpt-save-path <HF_OUTPUT_DIR>

The script does not apply KV cache quantization. The quantized checkpoint can then be inferenced via TensorRT-LLM's quickstart_advanced.py example script. This will produce different kinds of errors depending on the model and the configuration used.

For a dense Qwen3 8B model quantized to MXFP4 W4A8 I get this error:

[rank0]: RuntimeError: [TensorRT LLM Error][CutlassFp4GemmRunner][GEMM Dispatch] Arch unsupported for CUTLASS FP4 GEMM

Using NVFP4 (--use-nvfp4 option in the quantization script), leads to a weight loading error for the same dense model:

[TRT-LLM] [E] Error executing load_single_module with args ('model.layers.1.self_attn.o_proj', Linear(
  (lora): LoraLayer()
)): 'NoneType' object has no attribute 'to'

Note that the same errors appear if only the MLP block is quantized (--mlp-only option in the quantization script). When attempting to load an MXFP4 W4A8 quantized MoE Qwen3 30B-A3B model, on the other hand, the errors vary depending on the MoE backend. With CUTLASS, I get a segfault:

!!!!!!! Segfault encountered !!!!!!!
  File "<unknown>", line 0, in tensorrt_llm::_v1::torch_ext::FusedMoeRunner::setRunnerProfiles(std::optional<c10::ArrayRef<long> >)
  File "<unknown>", line 0, in tensorrt_llm::_v1::torch_ext::FusedMoeRunner::runMoe(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, at::Tensor const&, std::optional<at::Tensor> const&, at::Tensor const&, std::optional<at::Tensor> const&, std::optional<c10::ArrayRef<at::Tensor> > const&, std::optional<at::Tensor> const&, bool, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, long, long, long, long, long, long, bool, bool, std::optional<c10::ArrayRef<long> > const&, std::optional<long> const&, std::optional<long> const&, std::optional<long> const&, std::optional<at::Tensor> const&)

With TRTLLM, I get a clearer error suggesting lack of support:

[rank0]: RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: No kernel found for the given options: mDtypeA: MxE2m1, mDtypeB: E4m3, mDtypeC: E4m3, mUseDeepSeekFp8: 0, mTransposeMmaOutput: 1, mRouteAct: 1, mFusedAct: 1, mIsStaticBatch: 0, mTileSize: 8

I have tried using the TRITON backend as well, where I got other weight loading errors.

TL;DR: I'm wondering if MXFP4/NVFP4 W4A8 quantization is actually supported for general use (outside of specific cases like GPT-OSS) on Hopper and, if so, for which models and configurations specifically. Thanks in advance!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start by reproducing the reported cases with the ModelOpt script hf_mxfp4_ptq.py and TensorRT-LLM's quickstart_advanced.py on the H100 setup described. Compare the MXFP4 and NVFP4 results across dense and MoE models and the CUTLASS, TRTLLM, and TRITON backends. Done would require a confirmed support matrix or clearly scoped implementation and validation for the supported configurations.