intel / intel/llm-scaler

PyTorch 2.x runtime error

Open
#481 2 comments 0 reactions 1 assignee Claimed by @liu-shaojun View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

Docker image version: 0.14.0-b8.3
GPU: Intel Arc B70
Model: Qwen3.5-9B
Dtype: BFloat16
Runtime Error: (EngineCore_DP0 pid=181) ERROR 06-14 07:26:10 [core.py:936] RuntimeError: Inference tensors do not track version counter.
...
(EngineCore_DP0 pid=181) ERROR 06-14 07:26:10 [core.py:936] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 428, in forward_xpu
(EngineCore_DP0 pid=181) ERROR 06-14 07:26:10 [core.py:936] self.out_proj.weight.data = self.out_proj.weight.data.clone()

My workaround is to add " --enforce-eager".

Full Docker CMD: docker run -td --privileged --net=host --device=/dev/dri --shm-size="32g" --name vllm-arc-b70 \
-v /sys:/sys:ro \
-v /home/zhou80bin/Models/Qwen3.5:/llm/models/ \
-e HF_HUB_OFFLINE=1 \
--pull=never \
--entrypoint "" \
intel/llm-scaler-vllm:0.14.0-b8.3 \
vllm serve /llm/models/ --served-model-name Qwen3.5 --trust-remote-code \
--gpu-memory-utilization 0.85 --max-model-len 8192 --host 0.0.0.0 --port 8000 \
--enforce-eager

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.