[Bug]: GLM-5.1-FP8 produces corrupted output under tool-use on H200 (deepseek_v3.2 FP8 path)

System Info

• GPU: 8× NVIDIA H200 (SM90)
• TensorRT-LLM: 1.3.0rc16 (also reviewed rc17/rc18 changelogs; no relevant fix landed)
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc16
• Model: zai-org/GLM-5.1-FP8 (model_type glm_moe_dsa, arch GlmMoeDsaForCausalLM; served via DeepseekV3ForCausalLM)
• Quant: FP8 e4m3, block-scaled weights (weight_block_size [128,128]), dynamic activation; no KV-cache quant declared
• Serving: trtllm-serve, TP8 / EP8, PyTorch backend, moe_config.backend=CUTLASS, MTP off,
  --tool_parser glm47 --reasoning_parser deepseek-r1, kv_cache_config.dtype=auto

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Serve GLM-5.1-FP8 on 8×H200:

trtllm-serve zai-org/GLM-5.1-FP8 --backend pytorch --tp_size 8 --ep_size 8 \
  --tool_parser glm47 --reasoning_parser deepseek-r1 \
  --kv_cache_config '{"dtype":"auto","enable_block_reuse":true}' \
  --moe_config '{"backend":"CUTLASS"}'

Send a chat request that contains real function-tool definitions inside a large system
prompt (~7–8k tokens of agent/policy text + 3 simple function schemas):

POST /v1/chat/completions
{
  "model": "zai-org/GLM-5.1-FP8",
  "messages": [
    {"role":"system","content":"<~7–8k-token agent policy>"},
    {"role":"assistant","content":"Hi! How can I help you today?"},
    {"role":"user","content":"<a task that requires a tool call>"}
  ],
  "tools": [ <3 function schemas, simple string params, with descriptions> ],
  "tool_choice": "auto",
  "temperature": 0,
  "max_tokens": 1024,
  "chat_template_kwargs": {"enable_thinking": true}
}

Control (clean): the same request with the tools' description fields removed, or with 0 tools,
or with a small (<2048-token) system prompt.

Expected behavior

A coherent response and/or a structured tool_calls entry

actual behavior

The model degenerates: finish_reason="length" (never emits a stop token, runs to max_tokens),
tool_calls=[], and the generated text is incoherent — repetitive/random tokens, digit and https-heavy fragments, occasionally mixed-language.
Example output (greedy):

<arg_value>,https,2hat, the15|35ions,16#51Obn2247777735'170044https,720c523722|422 to>
... d46aining is_pn4601ed ... 7771https17https_n7ar4#U29n60

additional notes

We narrowed this down and believe it is FP8 numerics on the deepseek_v3.2 path, not config:

• Deterministic: reproduces at temperature=0 (greedy) → not sampling.
• Logits collapsed: logprobs are flat from the first generated token (top token ~7%,
  whole top-5 tiny) → saturated/degenerate logits.
• Cross-engine: same checkpoint serves coherently on SGLang/vLLM → not the weights.
• Not KV-cache FP8: with dtype=auto the checkpoint declares no kv_cache_quant_algo, so the
  main KV cache is bf16 (confirmed: allocated 110,448 B/token ≈ the bf16 figure, ~2× the fp8 figure;
  forcing dtype=bfloat16 is rejected — only fp8/nvfp4/auto accepted). MoE backend is CUTLASS.
• Not dispatch/RoPE: rc16 renames glm_moe_dsa→deepseek_v32 before model build, so the DSA
  attention path is selected; the config has no rope_scaling and the prompt is well within
  max_position_embeddings (202752).
• Not the tokenizer: all GLM special tokens encode atomically; chat template renders with tools
  and round-trips.
• Trigger: real tool descriptions in a large context (>~2048 tokens, i.e. the DSA sparse
  regime). Small/toolless prompts are (mostly) clean.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start by reproducing the trtllm-serve command with the DeepseekV3ForCausalLM/deepseek_v32 path, CUTLASS MoE backend, and the large tool-description prompt. Trace first-token logits through the DSA sparse attention and FP8 path, comparing the trigger with the clean controls; done means coherent generation with a structured tool_calls entry and a stop finish_reason.