NaN/inf logits on Qwen3.6-35B-A3B (Qwen3-Next) int4 + XPU graph at long context (Battlemage B60 & B70)
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
## Summary
On `intel/llm-scaler-vllm:0.21.0-b2`, serving **Qwen/Qwen3.6-35B-A3B** with online `sym_int4` + XPU
torch.compile graph (`VLLM_XPU_ENABLE_XPU_GRAPH=1`, `enforce_eager=False`, PIECEWISE) produces **NaN/inf
logits** on prompts that exceed the compile range, yielding either repeated-single-token garbage
(`!!!!…`, HTTP 200) or — when `logprobs` is requested — an HTTP 400 `"Out of range float values are not
JSON compliant"` (the NaN/inf logprob fails JSON serialization). It is **intermittent and correctness-
critical**: a NaN-perturbed forward pass can also yield fluent-but-wrong text that a repeat/entropy filter
won't catch. Reproduced on **both Arc Pro B60 and B70** (Battlemage), so it is SKU-independent.
## Environment
- Image: `intel/llm-scaler-vllm:0.21.0-b2`
- vLLM `0.21.1.dev0+gad7125a43.d20260802`, **`vllm-xpu-kernels 0.1.8.3.dev0+g3cab97a.d20260802`**, `torch 2.11.0+xpu`, libsycl.so.8 (oneAPI 2025)
- GPUs: Intel Arc Pro B60 (24 GB) and Arc Pro B70 (32 GB), `xe` driver
- Model: `Qwen/Qwen3.6-35B-A3B` (Qwen3-Next hybrid Mamba/GDN MoE, VL), text-only serving
## Repro
```bash
vllm serve --served-model-name Qwen3.6-35B-A3B \
--dtype float16 --quantization sym_int4 --allow-deprecated-quantization \
--kv-cache-dtype auto --trust-remote-code --limit-mm-per-prompt '{"image":0,"video":0}' \
--gpu-memory-util 0.9 --max-model-len 32768 -tp 1 \
--compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16]}'
# (VLLM_XPU_ENABLE_XPU_GRAPH=1)
```
Send a prompt > the compile range (default `max_num_batched_tokens`=2048), e.g. ~3,500 tokens, with
`"logprobs":1, "temperature":0`. A fraction return **HTTP 400 "Out of range float values are not JSON
compliant"** (NaN/inf logits). Rate rises with prompt length: ~**3–7/12 at ~3.5k tokens, ~5/6 at ~25k
tokens**. Below ~2048 tokens: clean.
## What we ruled out (measured)
- **Not the compile-range boundary.** Raising `--max-num-batched-tokens 32768` (which extends
`compile_ranges_endpoints` to cover the full context) makes it **worse** (7/12 at 3.5k). So the NaN is in
the compiled/graph **kernel** at long token counts, not an uncompiled fall-off path.
- **`--enforce-eager` is fully clean** (0 bad across 18 long-context samples incl. ~25k tokens) — but loses
the ~2–5× graph speedup, which is the whole value of the Arc serving path.
## Likely fix (why we're filing)
`0.21.0-b2` bundles **vllm-xpu-kernels 0.1.8.3**, which predates upstream `vllm-project/vllm-xpu-kernels`
**v0.1.12.1** (2026-07-30) and its directly-relevant changes:
- **#402 "[GDN] Split gdn attention into conv1d and gated_delta_rule"** — GDN is the prime suspect (Qwen3-Next hybrid).
- **#437 "fix low accuracy of Qwen3.5-27B/Qwen3.6-27B"** — a numerics/accuracy fix for this exact family.
- #479 attention `softmax_lse` layout alignment with CUDA FlashAttention.
We could not test the fix directly: the upstream v0.1.12.1 wheel links **libsycl.so.9 (oneAPI 2026)** while
b2's torch 2.11+xpu links **libsycl.so.8 (oneAPI 2025)** — incompatible in one process. **Request: please
ship v0.1.12.1 (or at least #402/#437) in the next `llm-scaler-vllm` image built against the matching
oneAPI/torch-xpu stack**, so int4 + graph is correct at long context without `enforce_eager`.
## Impact
This is the difference between the Arc int4+graph path being production-usable for real-length prompts
(clinical/EHR notes, RAG) vs. having to fall back to `enforce_eager` (~2.3× slower) for correctness.
Contributor guide
Assessment
This issue has not been assessed yet.