Failure to run nvfp4 GLM 4.5

System Info

B200

Who can help?

@juney-nvidia @Tracin

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Using https://github.com/deepinfra/TensorRT-LLM/tree/glm-jul28 (forked from upstream main 7/28), works with GLM-4.5-Air BF16 version correctly

Steps:

1. Quantize FP4 GLM-4.5-Air (in modelopt docker)

huggingface-cli download zai-org/GLM-4.5-Air
HF_PATH=./zai-org/GLM-4.5-Air
cd examples/llm_ptq
scripts/huggingface_example.sh --model $HF_PATH --quant nvfp4 --export_fmt hf

2. Run trtllm-serve (in tensorrt_llm docker)

export NCCL_NVLS_ENABLE=0; trtllm-serve /data/weights/vllm-zai-org--GLM-4.5-Air-FP4 --tp_size=8 --host=0.0.0.0 --port=8000 --backend=pytorch --max_num_tokens=4000 --max_seq_len 3000 --ep_size=8 --pp_size=1 --max_batch_size=1 --extra_llm_api_options extra-llm-api-config.yml

If I try to do the same steps with the GLM-4.5 none Air version, it hangs in one of the transformer layers. Even with BF16 weights. (Note, I also disabled the w1_weight_scale_2 != w3_weight_scale_2 check)

extra_llm_api_config.yml

enable_attention_dp: false
max_batch_size: 1
max_num_tokens: 4000
max_seq_len: 3000
kv_cache_config:
  free_gpu_memory_fraction: 0.8
enable_chunked_prefill: true
print_iter_log: true
disable_overlap_scheduler: true

Expected behavior

NVFP4 ckpt converted from model opt should work

actual behavior

 File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_deepseekv3.py", line 1247, in load_weights
    _load_weights_impl(self, weights, params_map={
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_utils.py", line 711, in _load_weights_impl
    module.load_weights(weights=[module_weights])
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py", line 437, in load_weights
    self.quant_method.load_weights(self, weights, self.weight_loading_mode)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/fused_moe/quantization.py", line 128, in load_weights
    self.load_quant_scales(module, weights)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/fused_moe/quantization.py", line 989, in load_quant_scales
    self.load_all_fp4_weight_scales_and_alphas(
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/fused_moe/quantization.py", line 947, in load_all_fp4_weight_scales_and_alphas
    self.load_expert_fc31_alpha_nvfp4(w1_weight_scale_2,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/fused_moe/quantization.py", line 897, in load_expert_fc31_alpha_nvfp4
    assert torch.allclose(
           ^^^^^^^^^^^^^^^
AssertionError: w1_weight_scale_2 != w3_weight_scale_2

additional notes

None

Start with the quantization and weight-loading paths in _torch/modules/fused_moe/quantization.py and _torch/models/modeling_deepseekv3.py, using the reported NVFP4 conversion and trtllm-serve commands to reproduce the failure. Compare GLM-4.5-Air with GLM-4.5 and investigate the w1_weight_scale_2 versus w3_weight_scale_2 assertion; done means the converted GLM-4.5 NVFP4 checkpoint loads and runs without hanging.