Could not run Qwen/Qwen3.5-27B-GPTQ-Int4 with release 0.14.0-b8.1
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
I tried the following commands and could not serve the model:
```
export MODEL_NAME="/llm/hub/models--Qwen--Qwen3.5-27B-GPTQ-Int4/snapshots/507bda6fcfcb5d3de0fe815d9e755bfeb58822e7/"
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
--model $MODEL_NAME \
--served-model-name Qwen3.5-27B \
--dtype=auto \
--enforce-eager \
--port 8000 \
--host 0.0.0.0 \
--trust-remote-code \
--disable-sliding-window \
--gpu-memory-util=0.9 \
--max-num-batched-tokens=2048 \
--disable-log-requests \
--max-model-len=8192 \
--block-size 64 \
-tp=2
```
Outputs:
```
[W313 12:20:53.685685090 OperatorEntry.cpp:208] Warning: Warning only once for all operators, other operators may also be overridden.
Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
registered at /pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
dispatch key: XPU
previous kernel: registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:36
new kernel: registered at /root/workspace/frameworks.ai.pytorch.ipex-gpu/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:172 (function operator())
WARNING 03-13 12:20:56 [_logger.py:68] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
(APIServer pid=1022) INFO 03-13 12:20:56 [api_server.py:1272] vLLM API server version 0.14.1.dev0+gb17039bcc.d20260311
(APIServer pid=1022) INFO 03-13 12:20:56 [utils.py:263] non-default args: {'model_tag': '/llm/hub/models--Qwen--Qwen3.5-27B-GPTQ-Int4/snapshots/507bda6fcfcb5d3de0fe815d9e755bfeb58822e7/', 'host': '0.0.0.0', 'model': '/llm/hub/models--Qwen--Qwen3.5-27B-GPTQ-Int4/snapshots/507bda6fcfcb5d3de0fe815d9e755bfeb58822e7/', 'trust_remote_code': True, 'max_model_len': 8192, 'enforce_eager': True, 'disable_sliding_window': True, 'served_model_name': ['Qwen3.5-27B'], 'tensor_parallel_size': 2, 'block_size': 64, 'max_num_batched_tokens': 2048}
(APIServer pid=1022) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1022) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1022) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(APIServer pid=1022) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(APIServer pid=1022) INFO 03-13 12:20:56 [model.py:533] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1022) INFO 03-13 12:20:56 [model.py:1549] Using max model len 8192
(APIServer pid=1022) Traceback (most recent call last):
(APIServer pid=1022) File "/usr/local/bin/vllm", line 8, in
(APIServer pid=1022) sys.exit(main())
(APIServer pid=1022) ^^^^^^
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=1022) args.dispatch_function(args)
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 60, in cmd
(APIServer pid=1022) uvloop.run(run_server(args))
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1022) return __asyncio.run(
(APIServer pid=1022) ^^^^^^^^^^^^^^
(APIServer pid=1022) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=1022) return runner.run(main)
(APIServer pid=1022) ^^^^^^^^^^^^^^^^
(APIServer pid=1022) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1022) return self._loop.run_until_complete(task)
(APIServer pid=1022) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1022) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1022) return await main
(APIServer pid=1022) ^^^^^^^^^^
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1319, in run_server
(APIServer pid=1022) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1338, in run_server_worker
(APIServer pid=1022) async with build_async_engine_client(
(APIServer pid=1022) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1022) return await anext(self.gen)
(APIServer pid=1022) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 173, in build_async_engine_client
(APIServer pid=1022) async with build_async_engine_client_from_engine_args(
(APIServer pid=1022) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1022) return await anext(self.gen)
(APIServer pid=1022) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 199, in build_async_engine_client_from_engine_args
(APIServer pid=1022) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=1022) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1374, in create_engine_config
(APIServer pid=1022) model_config = self.create_model_config()
(APIServer pid=1022) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1227, in create_model_config
(APIServer pid=1022) return ModelConfig(
(APIServer pid=1022) ^^^^^^^^^^^^
(APIServer pid=1022) File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__
(APIServer pid=1022) s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=1022) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=1022) Value error, ('The quantization method %s is deprecated and will be removed in future versions of vLLM. To bypass, set `--allow-deprecated-quantization`.', 'ipex') [type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]
(APIServer pid=1022) For further information visit https://errors.pydantic.dev/2.12/v/value_error
```
Contributor guide
Assessment
This issue has not been assessed yet.