intel/llm-scaler-vllm:0.21.0-b3.1 failing on Qwen 3.8 27b int4 AutoRound models

**Problem:**

I'm unable to run any of the Qwen 3.8 27b Int4 Auto Round models available in HF - when the chat server is sent a query, it errors out referencing /lib/x86_64-linux-gnu/libc.so.6

**Version:**

intel/llm-scaler-vllm:0.21.0-b3.1

**Model:**

**Frozenlock/Qwen3.8-27b-int4-AutoRound**

**Host OS:**

Ubuntu 26.04

**Video Card**

Intel B70 (Single)

**Compose File**

```
services:
llm-scaler-vllm:
image: intel/llm-scaler-vllm:0.21.0-b3.1
container_name: llm-scaler-qwen3827b
ports: ["8080:8080"]
devices: ["/dev/dri:/dev/dri"]
group_add: ["video", "991"] # your render-node GID; check `ls -l /dev/dri`
ipc: host
privileged: true
shm_size: '32gb'
volumes:
- type: bind
source: /data/llm/models
target: /models
environment:
- HF_HOME=/models
- VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
- VLLM_WORKER_MULTIPROC_METHOD=spawn
- VLLM_XPU_ENABLE_XPU_GRAPH=1
command:
- "--model"
- "Frozenlock/Qwen3.8-27b-int4-AutoRound"
- "--served-model-name"
- "Qwen/Qwen3.8-27B"
- "--port"
- "8080"
- "--host"
- "0.0.0.0"
- "--quantization"
- "auto_round"
- "--enable-prefix-caching"
- "--reasoning-parser"
- "qwen3"
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "qwen3_coder"
- "--enforce-eager"
- "--trust-remote-code"
- "--tensor-parallel-size"
- "1"
- "--gpu-memory-utilization"
- "0.9"
- "--max-model-len"
- "65536"
- "--max-num-batched-tokens"
- "8192"
- "--speculative-config"
- "{\"method\": \"mtp\", \"num_speculative_tokens\": 2}"
- "--default-chat-template-kwargs"
- "{\"preserve_thinking\": true, \"reasoning_effort\": \"medium\"}"
- "--generation-config"
- "vllm"
- "--override-generation-config"
- "{\"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"min_p\": 0.0, \"presence_penalty\": 0.0, \"repetition_penalty\": 1.0}"
restart: unless-stopped
```

Logs:

```
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, num_skipped_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.08196721311475408, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=520, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] EngineCore encountered a fatal error.
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] Traceback (most recent call last):
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1134, in run_engine_core
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] engine_core.run_busy_loop()
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1175, in run_busy_loop
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] self._process_engine_step()
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1214, in _process_engine_step
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] outputs, model_executed = self.step_fn()
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 527, in step_with_batch_queue
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] model_output = future.result()
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self.__get_result()
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] raise self._exception
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 97, in collective_rpc
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] result = run_method(self.driver_worker, method, args, kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 780, in sample_tokens
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self.model_runner.sample_tokens(grammar_output)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4325, in sample_tokens
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] propose_draft_token_ids(sampled_token_ids)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4286, in propose_draft_token_ids
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] self._draft_token_ids = self.propose_draft_token_ids(
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4869, in propose_draft_token_ids
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] draft_token_ids = self.drafter.propose(
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/spec_decode/llm_base_proposer.py", line 614, in propose
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ret_hidden_states = self.model(**model_kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 507, in __call__
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self.forward(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5_mtp.py", line 465, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] hidden_states = self.model(
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 507, in __call__
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self.forward(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5_mtp.py", line 186, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] hidden_states, residual = self.layers[current_step_idx](
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self._call_impl(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return forward_call(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1504, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] self.self_attn.forward(
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 887, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] attn_output = self.attn(q, k, v)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self._call_impl(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return forward_call(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/attention/attention.py", line 493, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] torch.ops.vllm.unified_attention_with_output(
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self._op(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py", line 40, in wrapper
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/attention/attention.py", line 723, in unified_attention_with_output
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] self.impl.forward(
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 925, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] return self._inner_forward(layer, query, key, value, kv_cache,
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 1136, in _inner_forward
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] eagle_ops.page_attn_decode(
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] RuntimeError: Expected query.scalar_type() == torch::kHalf to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] Exception raised from page_attn_decode at /src/custom-esimd-kernels-vllm/csrc/eagle/eagle.sycl:444 (most recent call first):
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x9d (0x7af9cecfa05d in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x69 (0x7af9cec8e7d3 in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #2: page_attn_decode(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor&, long, long, double, double) + 0x8c7 (0x7af71f0d2487 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #3: + 0x320f2 (0x7af71f0f00f2 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #4: + 0x31fc8 (0x7af71f0effc8 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #5: + 0x2ada5 (0x7af71f0e8da5 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #6: VLLM::EngineCore() [0x5821ef]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #7: _PyObject_MakeTpCall + 0x75 (0x5492b5 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #8: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #9: + 0xa48116 (0x7af9cdef8116 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #10: + 0xdbe245 (0x7af9ce26e245 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #11: + 0x64ed8fc (0x7af9be3ef8fc in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #12: + 0xb119c4 (0x7af9cdfc19c4 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #13: + 0xb11ec8 (0x7af9cdfc1ec8 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #14: torch::jit::_get_operation_for_overload_or_packet(std::vector, std::allocator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional) + 0x38 (0x7af9cdfc21d8 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #15: + 0x9f5afd (0x7af9cdea5afd in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #16: + 0x402840 (0x7af9cd8b2840 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #17: VLLM::EngineCore() [0x5821ef]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #18: PyObject_Call + 0x9c (0x54b45c in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #19: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #20: _PyObject_Call_Prepend + 0x18a (0x54abca in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #21: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #22: _PyObject_MakeTpCall + 0x13e (0x54937e in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #23: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #24: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #25: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #26: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #27: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #28: _PyObject_Call_Prepend + 0xc2 (0x54ab02 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #29: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #30: _PyObject_MakeTpCall + 0x75 (0x5492b5 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #31: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #32: VLLM::EngineCore() [0x54ceb4]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #33: PyObject_Call + 0x119 (0x54b4d9 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #34: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #35: VLLM::EngineCore() [0x54ceb4]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #36: PyObject_Call + 0x119 (0x54b4d9 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #37: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #38: _PyObject_Call_Prepend + 0x18a (0x54abca in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #39: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #40: _PyObject_MakeTpCall + 0x13e (0x54937e in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #41: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #42: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #43: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #44: _PyObject_Call_Prepend + 0xc2 (0x54ab02 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #45: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #46: _PyObject_MakeTpCall + 0x75 (0x5492b5 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #47: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #48: VLLM::EngineCore() [0x54ceb4]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #49: PyObject_Call + 0x119 (0x54b4d9 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #50: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #51: _PyObject_Call_Prepend + 0x18a (0x54abca in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #52: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #53: PyObject_Call + 0x9c (0x54b45c in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #54: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #55: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #56: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #57: PyEval_EvalCode + 0x15b (0x5d58bb in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #58: PyRun_StringFlags + 0xd3 (0x608a33 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #59: PyRun_SimpleStringFlags + 0x3e (0x6b426e in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #60: Py_RunMain + 0x481 (0x6bcf31 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #61: Py_BytesMain + 0x2d (0x6bc94d in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143] frame #62: + 0x2a1ca (0x7af9cf5af1ca in /lib/x86_64-linux-gnu/libc.so.6)
llm-scaler-qwen3827b | (EngineCore pid=353) ERROR 08-20 18:40:17 [core.py:1143]
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] AsyncLLM output_handler failed.
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] Traceback (most recent call last):
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] outputs = await engine_core.get_output_async()
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1005, in get_output_async
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] raise self._format_exception(outputs) from None
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] Error in chat completion stream generator.
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] Traceback (most recent call last):
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 504, in chat_completion_stream_generator
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] async for res in result_generator:
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 579, in generate
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] out = q.get_nowait() or await q.get()
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] ^^^^^^^^^^^^^
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/output_processor.py", line 88, in get
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] raise output
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] outputs = await engine_core.get_output_async()
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1005, in get_output_async
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] raise self._format_exception(outputs) from None
llm-scaler-qwen3827b | (APIServer pid=1) ERROR 08-20 18:40:17 [serving.py:1005] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
llm-scaler-qwen3827b | (EngineCore pid=353) Process EngineCore:
llm-scaler-qwen3827b | (EngineCore pid=353) Traceback (most recent call last):
llm-scaler-qwen3827b | (EngineCore pid=353) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
llm-scaler-qwen3827b | (EngineCore pid=353) self.run()
llm-scaler-qwen3827b | (EngineCore pid=353) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
llm-scaler-qwen3827b | (EngineCore pid=353) self._target(*self._args, **self._kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1145, in run_engine_core
llm-scaler-qwen3827b | (EngineCore pid=353) raise e
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1134, in run_engine_core
llm-scaler-qwen3827b | (EngineCore pid=353) engine_core.run_busy_loop()
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1175, in run_busy_loop
llm-scaler-qwen3827b | (EngineCore pid=353) self._process_engine_step()
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1214, in _process_engine_step
llm-scaler-qwen3827b | (EngineCore pid=353) outputs, model_executed = self.step_fn()
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 527, in step_with_batch_queue
llm-scaler-qwen3827b | (EngineCore pid=353) model_output = future.result()
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
llm-scaler-qwen3827b | (EngineCore pid=353) return self.__get_result()
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
llm-scaler-qwen3827b | (EngineCore pid=353) raise self._exception
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 97, in collective_rpc
llm-scaler-qwen3827b | (EngineCore pid=353) result = run_method(self.driver_worker, method, args, kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
llm-scaler-qwen3827b | (EngineCore pid=353) return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
llm-scaler-qwen3827b | (EngineCore pid=353) return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 780, in sample_tokens
llm-scaler-qwen3827b | (EngineCore pid=353) return self.model_runner.sample_tokens(grammar_output)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
llm-scaler-qwen3827b | (EngineCore pid=353) return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4325, in sample_tokens
llm-scaler-qwen3827b | (EngineCore pid=353) propose_draft_token_ids(sampled_token_ids)
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4286, in propose_draft_token_ids
llm-scaler-qwen3827b | (EngineCore pid=353) self._draft_token_ids = self.propose_draft_token_ids(
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4869, in propose_draft_token_ids
llm-scaler-qwen3827b | (EngineCore pid=353) draft_token_ids = self.drafter.propose(
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/spec_decode/llm_base_proposer.py", line 614, in propose
llm-scaler-qwen3827b | (EngineCore pid=353) ret_hidden_states = self.model(**model_kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 507, in __call__
llm-scaler-qwen3827b | (EngineCore pid=353) return self.forward(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5_mtp.py", line 465, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) hidden_states = self.model(
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 507, in __call__
llm-scaler-qwen3827b | (EngineCore pid=353) return self.forward(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5_mtp.py", line 186, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) hidden_states, residual = self.layers[current_step_idx](
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) return self._call_impl(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) return forward_call(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 1504, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) self.self_attn.forward(
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_next.py", line 887, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) attn_output = self.attn(q, k, v)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) return self._call_impl(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
llm-scaler-qwen3827b | (EngineCore pid=353) return forward_call(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/attention/attention.py", line 493, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) torch.ops.vllm.unified_attention_with_output(
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/torch/_ops.py", line 1269, in __call__
llm-scaler-qwen3827b | (EngineCore pid=353) return self._op(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py", line 40, in wrapper
llm-scaler-qwen3827b | (EngineCore pid=353) return func(*args, **kwargs)
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/attention/attention.py", line 723, in unified_attention_with_output
llm-scaler-qwen3827b | (EngineCore pid=353) self.impl.forward(
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 925, in forward
llm-scaler-qwen3827b | (EngineCore pid=353) return self._inner_forward(layer, query, key, value, kv_cache,
llm-scaler-qwen3827b | (EngineCore pid=353) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-scaler-qwen3827b | (EngineCore pid=353) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 1136, in _inner_forward
llm-scaler-qwen3827b | (EngineCore pid=353) eagle_ops.page_attn_decode(
llm-scaler-qwen3827b | (EngineCore pid=353) RuntimeError: Expected query.scalar_type() == torch::kHalf to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)
llm-scaler-qwen3827b | (EngineCore pid=353) Exception raised from page_attn_decode at /src/custom-esimd-kernels-vllm/csrc/eagle/eagle.sycl:444 (most recent call first):
llm-scaler-qwen3827b | (EngineCore pid=353) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x9d (0x7af9cecfa05d in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x69 (0x7af9cec8e7d3 in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #2: page_attn_decode(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor&, long, long, double, double) + 0x8c7 (0x7af71f0d2487 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #3: + 0x320f2 (0x7af71f0f00f2 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #4: + 0x31fc8 (0x7af71f0effc8 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #5: + 0x2ada5 (0x7af71f0e8da5 in /opt/venv/lib/python3.12/site-packages/custom_esimd_kernels_vllm/eagle_ops.cpython-312-x86_64-linux-gnu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #6: VLLM::EngineCore() [0x5821ef]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #7: _PyObject_MakeTpCall + 0x75 (0x5492b5 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #8: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #9: + 0xa48116 (0x7af9cdef8116 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #10: + 0xdbe245 (0x7af9ce26e245 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #11: + 0x64ed8fc (0x7af9be3ef8fc in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #12: + 0xb119c4 (0x7af9cdfc19c4 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #13: + 0xb11ec8 (0x7af9cdfc1ec8 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #14: torch::jit::_get_operation_for_overload_or_packet(std::vector, std::allocator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional) + 0x38 (0x7af9cdfc21d8 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #15: + 0x9f5afd (0x7af9cdea5afd in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #16: + 0x402840 (0x7af9cd8b2840 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #17: VLLM::EngineCore() [0x5821ef]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #18: PyObject_Call + 0x9c (0x54b45c in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #19: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #20: _PyObject_Call_Prepend + 0x18a (0x54abca in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #21: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #22: _PyObject_MakeTpCall + 0x13e (0x54937e in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #23: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #24: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #25: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #26: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #27: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #28: _PyObject_Call_Prepend + 0xc2 (0x54ab02 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #29: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #30: _PyObject_MakeTpCall + 0x75 (0x5492b5 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #31: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #32: VLLM::EngineCore() [0x54ceb4]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #33: PyObject_Call + 0x119 (0x54b4d9 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #34: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #35: VLLM::EngineCore() [0x54ceb4]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #36: PyObject_Call + 0x119 (0x54b4d9 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #37: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #38: _PyObject_Call_Prepend + 0x18a (0x54abca in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #39: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #40: _PyObject_MakeTpCall + 0x13e (0x54937e in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #41: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #42: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #43: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #44: _PyObject_Call_Prepend + 0xc2 (0x54ab02 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #45: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #46: _PyObject_MakeTpCall + 0x75 (0x5492b5 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #47: _PyEval_EvalFrameDefault + 0xa89 (0x5d7389 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #48: VLLM::EngineCore() [0x54ceb4]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #49: PyObject_Call + 0x119 (0x54b4d9 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #50: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #51: _PyObject_Call_Prepend + 0x18a (0x54abca in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #52: VLLM::EngineCore() [0x5a37f8]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #53: PyObject_Call + 0x9c (0x54b45c in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #54: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #55: VLLM::EngineCore() [0x54cded]
llm-scaler-qwen3827b | (EngineCore pid=353) frame #56: _PyEval_EvalFrameDefault + 0x4c32 (0x5db532 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #57: PyEval_EvalCode + 0x15b (0x5d58bb in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #58: PyRun_StringFlags + 0xd3 (0x608a33 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #59: PyRun_SimpleStringFlags + 0x3e (0x6b426e in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #60: Py_RunMain + 0x481 (0x6bcf31 in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #61: Py_BytesMain + 0x2d (0x6bc94d in VLLM::EngineCore)
llm-scaler-qwen3827b | (EngineCore pid=353) frame #62: + 0x2a1ca (0x7af9cf5af1ca in /lib/x86_64-linux-gnu/libc.so.6)
llm-scaler-qwen3827b | (EngineCore pid=353)
llm-scaler-qwen3827b | (APIServer pid=1) INFO: 192.168.1.1:59237 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
llm-scaler-qwen3827b | (APIServer pid=1) INFO: Shutting down
llm-scaler-qwen3827b | (APIServer pid=1) INFO: Waiting for application shutdown.
llm-scaler-qwen3827b | (APIServer pid=1) INFO: Application shutdown complete.
llm-scaler-qwen3827b | (APIServer pid=1) INFO: Finished server process [1]
``

Reproduce the chat request with the listed Docker Compose configuration, model, and image version. Start at the traceback entry points in vllm/v1/worker/gpu_model_runner.py, vllm/v1/spec_decode/llm_base_proposer.py, and the Qwen model files qwen3_5_mtp.py and qwen3_next.py. Done means the Qwen 3.8 27B AutoRound model handles a chat query without the fatal libc-related error.

Written by the indexing model from the issue text.