intel / intel/llm-scaler

torch._dynamo.exc.Unsupported: _has_xmx crash when running 32B model on single Arc B60 Pro

Open
#325 1 comment 0 reactions 1 assignee Claimed by @gc-fu View on GitHub
vllm
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

**Image:** intel/llm-scaler-vllm:latest (45.8GB) and intel/llm-scaler-vllm:0.14.0-b8.1 (48.3GB) — both affected

**Hardware:** Intel Arc Pro B60 (single card, 24GB), Ubuntu 24.04 LTS, kernel 6.17, xe driver

**Model:** Qwen/Qwen3-32B-AWQ

**Command:**
docker run --rm \
--device /dev/dri:/dev/dri \
--device /dev/dri/by-path:/dev/dri/by-path \
--group-add [render GID] --group-add [video GID] \
-v /home/todd/models:/models \
-v /home/todd/vllm-cache:/root/.cache/vllm \
-e VLLM_TARGET_DEVICE=xpu \
--entrypoint "" \
intel/llm-scaler-vllm:latest \
vllm serve /models/Qwen3-32B-AWQ --dtype bfloat16

**Error (during compilation phase, after model shards load):**
torch._dynamo.exc.Unsupported: call_function ipex._C._has_xmx in skip_files
File ".../intel_extension_for_pytorch/transformers/optimize.py"
and ipex._C._has_xmx(0)

**Root cause:**
ipex._C._has_xmx is a C++ pybind function. torch.dynamo cannot trace it,
but it is NOT registered in allow_in_graph unlike the other IPEX C++ functions.

In intel_extension_for_pytorch/frontend.py, these are registered:
torch._dynamo.allow_in_graph(_IPEXLinear)
torch._dynamo.allow_in_graph(_IPEXConv1d)
... etc

But ipex._C._has_xmx is NOT registered, so dynamo crashes when it tries
to trace through the quantization path in transformers/optimize.py.

**Fix (one line, in frontend.py):**
torch._dynamo.allow_in_graph(ipex._C._has_xmx)
torch._dynamo.allow_in_graph(ipex._C._has_2d_block_array) # same issue

**Workaround:** --enforce-eager bypasses compilation but gives ~3 t/s on
32B (no XMX engagement). Not a viable solution for 32B inference.

**Note:** Model loads fine, Flash Attention initializes, crash only occurs
during the torch.compile/inductor step. With enforce-eager the model runs
but without XMX kernels (~3 t/s vs expected 20-40 t/s).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.