intel / intel/llm-scaler

[vllm] Pre-quantized FP8 Qwen3.6-27B-FP8 fails on XPU: missing PlatformEnum.XPU in FP8 block kernel table (0.21.0-b1); AttributeError at construction (0.14.0-b8.3.2)

Open
#550 4 comments 1 reaction 1 assignee Claimed by @gc-fu View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

## Environment

- 2x Intel Arc Pro B60 (BMG G21, 24 GB, PCI `8086:e211`), Alder Lake-S host
- Host: Ubuntu 24.04, kernel 7.0.0-28-generic (xe driver), host compute-runtime 25.18.33578.15
- Note: the two cards do **not** share an upstream PCIe bridge; the kernel refuses peer-to-peer DMA (`xe ... cannot be used for peer-to-peer DMA as the client and provider do not share an upstream bridge or whitelisted host bridge`). Workarounds used so multi-GPU works at all (may be worth documenting): `FI_PROVIDER=tcp`, `FI_TCP_IFACE=lo` (oneCCL OFI bootstrap otherwise spins forever inside Docker), `CCL_TOPO_P2P_ACCESS=0`, and for pipeline parallelism `CCL_SEND=direct CCL_RECV=direct` (otherwise `zeMemOpenIpcHandle` fails at the first PP send/recv).
- Model: official `Qwen/Qwen3.6-27B-FP8` (fine-grained FP8, block size 128)
- Serve command (both images):
`vllm serve /llm/models/Qwen3.6-27B-FP8 -tp 2 --kv-cache-dtype auto --max-model-len 8192 --gpu-memory-utilization 0.90`

## Issue 1 — `intel/llm-scaler-vllm:0.21.0-b1`: `KeyError: `

Model construction reaches the FP8 linear path of the GDN linear-attention layers
(`Qwen3_5DecoderLayer -> GatedDeltaNetAttention -> MergedColumnParallelLinear`) and dies in
`choose_scaled_mm_linear_kernel`:

```
File ".../vllm/model_executor/kernels/linear/__init__.py", line 355, in choose_scaled_mm_linear_kernel
for kernel in possible_kernels[current_platform._enum]:
KeyError:
```

`_POSSIBLE_FP8_BLOCK_KERNELS` (and `_POSSIBLE_FP8_KERNELS`) only contain `CUDA`, `ROCM` and `CPU` entries. Any model whose FP8 path goes through `init_fp8_linear_kernel` with per-group activation scales is unusable on XPU in this image.

### What we tried

Locally patching the image:

1. added `PlatformEnum.XPU: [TritonFp8BlockScaledMMKernel]` to `_POSSIBLE_FP8_BLOCK_KERNELS`;
2. relaxed `TritonFp8BlockScaledMMKernel.is_supported()` to accept XPU.

With that, the model **constructs and loads** (14.5 GiB/GPU, TP=2), and the kernel is selected
(`Selected TritonFp8BlockScaledMMKernel for Fp8LinearMethod`). However:

- default compile mode fails during inductor autotuning:
```
File ".../triton/backends/intel/compiler.py", line 385, in make_llir
pm.run(mod, 'make_llir')
RuntimeError: PassManager::run failed
torch._inductor.exc.InductorError: RuntimeError: Failed to run autotuning code block: PassManager::run failed
```
- with `--enforce-eager` the server comes up and answers correctly, but decode collapses to
~1.2 tok/s (prefill ~102 tok/s), which is not usable.

## Issue 2 — `intel/llm-scaler-vllm:0.14.0-b8.3.2`: AttributeError at construction

Same model/checkpoint, same serve command:

```
File ".../vllm/model_executor/models/qwen3_5.py", line 1108, in __init__
_qkvz_sz = self.linear_attn.in_proj_qkvz.weight.shape[0]
AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight'
```

Happens with and without `VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1`. So the pre-quantized FP8 checkpoint is not loadable on this image either.

## Issue 3 (minor) — 0.14.0-b8.3.2: dtype mismatch with runtime FP8 + bf16

The path that *does* work is the BF16 checkpoint (`Qwen/Qwen3.6-27B`) with `--quantization fp8`, but only with `--dtype float16`. With the default bf16 the warm-up crashes:

```
RuntimeError: expected self and mat2 to have the same dtype, but got: c10::Half != c10::BFloat16
```

i.e. an FP8 GEMM appears to produce fp16 regardless of model dtype. Since Qwen3.6 is trained in bf16 (and `mamba_ssm_dtype: float32`), having to force fp16 is not ideal.

## Working configuration (for reference)

`0.14.0-b8.3.2` + BF16 checkpoint + `--quantization fp8 --enforce-eager --dtype float16` + the CCL env above. Measured on 2x B60 (1024 in / 256 out): TP=2 ~15 tok/s single-stream, PP=2 ~107 tok/s aggregate at 16 streams. Output quality is coherent.

## Requests

1. Add an XPU entry (or an XPU-capable kernel) to the FP8 scaled-mm kernel tables so pre-quantized block-FP8 checkpoints (Qwen3.6-27B-FP8 and similar) work on XPU.
2. Fix the triton-xpu `PassManager::run failed` in inductor autotuning for this path, or document that eager is required.
3. Fix `qwen3_5.py` construction with pre-quantized FP8 on the 0.14 line.
4. Fix or document the bf16/fp16 dtype mismatch with runtime `--quantization fp8`.

Happy to provide full logs or run test builds on this dual-B60 host.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.