[Bug] Qwen3-Next GDN attention fails on XPU: hardcoded FP16 buffers mismatch BF16 dt_bias parameter

Summary

On intel/llm-scaler-vllm:0.14.0-b8.3.1, running any Qwen3-Next-architecture model (e.g. Qwen3-Coder-Next, Qwen3.5/3.6-A3B) in its native BF16 with online FP8 quantization fails at the first forward pass inside the Gated DeltaNet (GDN) attention kernel:

RuntimeError: dt_bias dtype must match core_attn_out dtype (float16/bfloat16),
but got dt_bias=BFloat16, core_attn_out=Half

The model loads fully and the server reaches "Application startup complete" — the crash is triggered by the first inference request.

Environment

Image: intel/llm-scaler-vllm:0.14.0-b8.3.1 (vLLM 0.14.1.dev0+gb17039bcc)
GPU: 4× Intel Arc Pro B70 (32 GB), TP=4
Model: Qwen3-Coder-Next (BF16 weights), --quantization fp8, --enforce-eager
OS: Ubuntu 26.04, kernel 7.0

Root cause

The XPU GDN kernel (torch.ops._xpu_C.gdn_attention) requires dt_bias and core_attn_out to share a dtype. They don't, because of an inconsistency inside vllm/model_executor/models/qwen3_next.py (line numbers below are from the stock 0.14.0-b8.3.1 file):

The pre-allocated GDN scratch buffers are hardcoded to torch.float16 throughout __init__ (lines 511, 519, 524, 529, 534, 540 for the BSZ=1 decode buffers and 559, 564, 569, 574, 578 for the BSZ>1 _m_* buffers). This includes the two buffers that become core_attn_out: _decode_attn_out_buf and _m_attn_out.
self.dt_bias is an nn.Parameter declared at line 475 (weight loader attached at 485) that inherits the model's configured dtype — BF16 when the model runs with its native --dtype bfloat16.
In forward_xpu (defined at line 672), core_attn_out is assigned from the hardcoded-FP16 buffers on the decode and small-batch paths:

pythonif is_decode:
core_attn_out = self._decode_attn_out_buf # line 763 — float16
elif num_tokens <= self._max_bsz:
core_attn_out = self._m_attn_out[:num_tokens] # line 766 — float16
else:
core_attn_out = torch.empty(..., dtype=hidden_states.dtype, ...) # line 771 — correct

The kernel is then called at line 835 with FP16 core_attn_out and dt_bias=self.dt_bias (BF16) passed at line 850, and rejects the mismatch.

Note that the large-batch fallback at step 3 (dtype=hidden_states.dtype, line 771) does the right thing — it respects the model dtype. Only the pre-allocated buffers ignore it. The same hardcoded-float16 pattern repeats in the alternate paths forward_xpu_with_precomputed_proj (line 936, buffer reuse at 946) and forward_xpu_batched_precomputed_proj (line 1027, allocation at 1050), so any fix should cover all three.

The codebase already shows partial awareness of this kernel's dtype fragility: the comment at line 684 references gdn_attention XE2 chunk kernel OOB writes in mixed-dtype conditions. The dtype contract for this kernel is evidently known to be brittle; this report is the concrete buffer-allocation cause.

Workaround (functional but suboptimal)

--dtype float16 forces the whole model to FP16, so dt_bias becomes FP16 and matches the hardcoded buffers. This runs:

Qwen3-Coder-Next, FP8, --dtype float16, --enforce-eager, TP=4, --max-model-len 8192 → ~25–27 t/s single-stream (single concurrent request, eager mode).

This is the same flag the reporter in #382 was already using; it sidesteps the dtype crash but is not a real fix — it forces FP16 on a BF16-native model and gives up the BF16 numeric range.

Why it can't be patched from Python

Attempts to cast dt_bias to match the FP16 buffers (so the model can stay BF16) all fail on XPU:

Per-call self.dt_bias.half() at the kernel call sites → hangs during warmup (per-forward cast appears to deadlock the XPU kernel queue).
Cached .half() copy created once in forward_xpu → kernel reports dt_bias=UNKNOWN_SCALAR; the XPU runtime cannot represent/cast an on-device BF16→FP16 copy here.

This points to the fix needing to be on the Intel side, not in user model code.

Proposed fix

Make the pre-allocated GDN buffers respect the configured model dtype instead of hardcoding torch.float16 — i.e. allocate _decode_attn_out_buf, _decode_z_buf, _m_attn_out, _m_z, etc. with dtype=config.torch_dtype (matching what the large-batch fallback already does with hidden_states.dtype).

Alternatively, make the _xpu_C.gdn_attention kernel accept mixed FP16/BF16 inputs (cast dt_bias internally to core_attn_out's dtype).

Related

#382 — same architecture/container, reporter is already on --dtype float16 and hits a downstream OOM; the dtype mismatch underneath was not diagnosed.
#339, #386 — other Qwen3-Next-family failures (hangs / OOM / MTP crash) that may share this root cause.

Upstream, Qwen3-Next officially requires vllm>=0.15.0. This image backports the architecture onto a 0.14.1 base; the incomplete GDN port is consistent with that. Rebuilding the image on a 0.15.x base (a maintainer-side change; nothing required from users beyond pulling the new tag) may pick up a more complete GDN path and resolve this without patching the backport.