intel / intel/llm-scaler

Start Qwen-27B vllm serving failed on B60

Open
#616 1 comment 0 reactions 1 assignee Claimed by @365614269 View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

The system has 32GB ram, the vllm cmd is below:
export VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=0
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
--model /llm/models/Qwen3.6-27B \
--served-model-name Qwen3.6-27B \
--dtype=float16 \
--enforce-eager \
--port 8000 \
--host 0.0.0.0 \
--trust-remote-code \
--disable-sliding-window \
--gpu-memory-util=0.8 \
--max-num-batched-tokens=8192 \
--max-model-len=131072 \
--block-size 64 \
--quantization sym_int4 \
-tp=1

The error log is
(EngineCore pid=291) Process EngineCore: 16:12:45 [62/152]
(EngineCore pid=291) Traceback (most recent call last):
(EngineCore pid=291) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=291) self.run()
(EngineCore pid=291) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=291) self._target(*self._args, **self._kwargs)
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1145, in run_engine_core
(EngineCore pid=291) raise e
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1115, in run_engine_core
(EngineCore pid=291) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=291) return func(*args, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 881, in __init__
(EngineCore pid=291) super().__init__(
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 118, in __init__
(EngineCore pid=291) self.model_executor = executor_class(vllm_config)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=291) return func(*args, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=291) self._init_executor()
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 60, in _init_executor
(EngineCore pid=291) self.driver_worker.init_device()
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 317, in init_device
(EngineCore pid=291) self.worker.init_device() # type: ignore
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/xpu_worker.py", line 90, in init_device
(EngineCore pid=291) torch.distributed.all_reduce(torch.zeros(1).xpu())
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
(EngineCore pid=291) return func(*args, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3068, in all_reduce
(EngineCore pid=291) work = group.allreduce([tensor], opts)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) RuntimeError: level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.