NVIDIA / NVIDIA/TensorRT-LLM

Qwen3-0.6B-FP8模型在使用quickstart_advanced.py时报错于 rmsnorm 数据类型 Float 无法调度

Open
#5,383 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

我在使用 TensorRT-LLM 0.20.0 版本运行 examples/pytorch/quickstart_advanced.py,加载模型 Qwen3-0.6B-FP8 进行推理时,遇到了如下错误:
RuntimeError: rmsnorm(...)::<lambda()> failed to dispatch data type Float

Qwen3-0.6B-FP8模型的safetensors 格式:
model.embed_tokens.weight: torch.float32
model.layers.0.input_layernorm.weight: torch.float32
model.layers.0.mlp.down_proj.weight: torch.float8_e4m3fn
model.layers.0.mlp.down_proj.weight_scale_inv: torch.float32
model.layers.0.mlp.gate_proj.weight: torch.float8_e4m3fn

复现过程:
docker run --rm -it --gpus all
-v ~/Projects/tensorrt_projects/TensorRT-LLM:/app/tensorrt_llm
-v ~/Projects/models:/models
tensorrt_llm:0.20.0
bash

python3 examples/pytorch/quickstart_advanced.py
--model_dir /models/qwen3-0.6b-fp8
--kv_cache_fraction 0.6

从huggingface上下载的qwen3-0.6b-fp8模型,挂载到0.20.0版本的容器中,运行quickstart_advanced.py,报错RuntimeError: rmsnorm(...)::<lambda()> failed to dispatch data type Float

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/pytorch/quickstart_advanced.py and reproduce the failure in the TensorRT-LLM 0.20.0 container using Qwen3-0.6B-FP8. Trace the rmsnorm call and the listed model tensor dtypes to determine why Float cannot be dispatched. Done means the quickstart runs inference with this model without the rmsnorm dispatch error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.