Qwen3-0.6B-FP8模型在使用quickstart_advanced.py时报错于 rmsnorm 数据类型 Float 无法调度
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
我在使用 TensorRT-LLM 0.20.0 版本运行 examples/pytorch/quickstart_advanced.py,加载模型 Qwen3-0.6B-FP8 进行推理时,遇到了如下错误:
RuntimeError: rmsnorm(...)::<lambda()> failed to dispatch data type Float
Qwen3-0.6B-FP8模型的safetensors 格式:
model.embed_tokens.weight: torch.float32
model.layers.0.input_layernorm.weight: torch.float32
model.layers.0.mlp.down_proj.weight: torch.float8_e4m3fn
model.layers.0.mlp.down_proj.weight_scale_inv: torch.float32
model.layers.0.mlp.gate_proj.weight: torch.float8_e4m3fn
复现过程:
docker run --rm -it --gpus all
-v ~/Projects/tensorrt_projects/TensorRT-LLM:/app/tensorrt_llm
-v ~/Projects/models:/models
tensorrt_llm:0.20.0
bash
python3 examples/pytorch/quickstart_advanced.py
--model_dir /models/qwen3-0.6b-fp8
--kv_cache_fraction 0.6
从huggingface上下载的qwen3-0.6b-fp8模型,挂载到0.20.0版本的容器中,运行quickstart_advanced.py,报错RuntimeError: rmsnorm(...)::<lambda()> failed to dispatch data type Float
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/pytorch/quickstart_advanced.py and reproduce the failure in the TensorRT-LLM 0.20.0 container using Qwen3-0.6B-FP8. Trace the rmsnorm call and the listed model tensor dtypes to determine why Float cannot be dispatched. Done means the quickstart runs inference with this model without the rmsnorm dispatch error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100