[Bug]: GLM-5.1-NVFP4 serve fails at warmup: PRE_MLP_FUSION accesses missing gate_up_proj.input_scale on excluded dense layers

System Info

• CPU architecture: x86_64
• GPU: 2x NVIDIA B300 (SM103), Kubernetes nvidia.com/gpu: 2
• TensorRT-LLM container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc17
• Model: nvidia/GLM-5.1-NVFP4 (mounted at /models, served as zai-org/GLM5.1-NVFP4)
• Parallelism: --tp_size 2 (single-node, 2 GPUs)
• OS: Linux (Kubernetes node)
• Deployment: trtllm-serve in a K8s Deployment with ConfigMap-mounted YAML config

nvidia-smi
python --version
pip show tensorrt_llm torch

Who can help?

@Tracin

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Serve nvidia/GLM-5.1-NVFP4 via Kubernetes using a ConfigMap for TRT-LLM YAML config and a 2-GPU Deployment.

ConfigMap (config.yml):

cuda_graph_config:
  enable_padding: true
  max_batch_size: 128
enable_attention_dp: false
enable_chunked_prefill: true
kv_cache_config:
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.75
  dtype: auto
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 1
stream_interval: 10
moe_config:
  backend: TRTLLM

Container entrypoint (simplified):

export LD_LIBRARY_PATH=/usr/local/tensorrt/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}
exec trtllm-serve /models \
    --host=0.0.0.0 \
    --port=8000 \
    --served_model_name=zai-org/GLM5.1-NVFP4 \
    --max_batch_size=128 \
    --max_num_tokens=8192 \
    --tp_size=2 \
    --config=/config/config.yml

Steps:

1. Mount the GLM-5.1-NVFP4 checkpoint at /models.
2. Mount the ConfigMap at /config/config.yml.
3. Deploy with 2 GPUs and start trtllm-serve as above.
4. Server fails during executor warmup before accepting requests.

Error (truncated):

AttributeError: 'Linear' object has no attribute 'input_scale'

  File ".../tensorrt_llm/_torch/models/modeling_deepseekv3.py", line 1551, in forward_mlp
    scale=self.mlp.gate_up_proj.input_scale,
  ...
RuntimeError: Executor worker returned error

Relevant checkpoint quant config (hf_quant_config.json from nvidia/GLM-5.1-NVFP4):

{
  "quantization": {
    "quant_algo": "NVFP4",
    "kv_cache_quant_algo": "FP8",
    "group_size": 16,
    "exclude_modules": [
      "lm_head",
      "model.layers.0*",
      "model.layers.1.*",
      "model.layers.2.*",
      "model.layers.3.mlp.shared_experts*",
      "model.layers.3.self_attn*",
      "..."
    ]
  }
}

Model config.json: first_k_dense_replace: 3 → layers 0/1/2 use dense GatedMLP.

Expected behavior

trtllm-serve should start successfully with nvidia/GLM-5.1-NVFP4 on 2x B300 using the config above (moe_config.backend: TRTLLM, MTP enabled, --tp_size 2).

For dense layers whose MLP weights are excluded from NVFP4 quantization (layers 0–2 in this checkpoint), the server should fall back to the non-fused RMSNorm + BF16 MLP path instead of crashing during warmup.

actual behavior

Server crashes during PyExecutor warmup with:

AttributeError: 'Linear' object has no attribute 'input_scale'

In DeepseekV3DecoderLayer.forward_mlp(), PRE_MLP_FUSION is enabled and the code reads self.mlp.gate_up_proj.input_scale for AllReduceFusionOp.RESIDUAL_RMS_NORM_QUANT_NVFP4. For excluded dense layers, gate_up_proj is an unquantized Linear (UnquantizedLinearMethod) with no input_scale.

Root cause: layer-level fusion setup does not match per-module quantization exclusion.

1. _get_decoder_layer_quant_config() only checks whether the whole layer name model.layers.{idx} is excluded. GLM-5.1-NVFP4 uses submodule patterns (model.layers.0*, etc.), so layer-level is_nvfp4 stays True.
2. With TP>1 and global NVFP4, dense layers set PRE_MLP_FUSION = True in DeepseekV3DecoderLayer.__init__.
3. Later, apply_quant_config_exclude_modules() correctly de-quantizes excluded modules such as model.layers.0.mlp.gate_up_proj.
4. At runtime, forward_mlp() still takes the NVFP4 fusion path and crashes.

fusion_config is per decoder layer (not model-global); only the first 3 dense layers hit this PRE_MLP_FUSION path.

Workaround:

export TRTLLM_DEEPSEEK_EAGER_FUSION_DISABLED=1

additional notes

• Affects models on the DeepSeek V3 / GLM MoE DSA path (GlmMoeDsaForCausalLM, DeepseekV32ForCausalLM), including modeling_glm.py and modeling_exaone_moe.py.
• Suggested fixes:
  1. Runtime reconcile: only use PRE_MLP NVFP4 fusion when gate_up_proj.has_nvfp4 and input_scale exist after apply_quant_config_exclude_modules().
  2. Init-time: when setting PRE_MLP_FUSION, also check quant_config.is_module_excluded_from_quantization(f"model.layers.{layer_idx}.mlp.gate_up_proj").
• Draft fix PR: https://github.com/NVIDIA/TensorRT-LLM/pull/15515
• Related deployment guide: docs/source/deployment-guide/deployment-guide-for-glm-5-on-trtllm.md

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start in tensorrt_llm/_torch/models/modeling_deepseekv3.py at DeepseekV3DecoderLayer.forward_mlp() and the layer initialization that enables PRE_MLP_FUSION. Then inspect _get_decoder_layer_quant_config() and apply_quant_config_exclude_modules(), including the related paths in modeling_glm.py and modeling_exaone_moe.py. Done means GLM-5.1-NVFP4 warmup succeeds with excluded dense-layer MLP modules, while the existing eager-fusion workaround remains unnecessary.