intel / intel/llm-scaler

[Regression 0.21.0-b1] Qwen3.6-27B gdn_attention XPU intermittently corrupts prefill; first token becomes "!" with identical greedy input

Open
#559 9 comments 0 reactions 1 assignee Claimed by @gc-fu View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

## Summary

Qwen3.6-27B on llm-scaler-vllm:0.21.0-b1 intermittently produces
`!!!!!!!!` from the first generated token.

This appears to be a regression or incomplete backport of #483, which was
reported as fixed in 0.14.0-b8.3.2.

The issue is independent of OpenCode and is reproducible through
`/v1/completions` using captured prompt token IDs.

## Controlled reproduction

Model:
`Qwen3.6-27B-int4-AutoRound`

Hardware:
- Intel Arc Pro B70 32 GB
- Single GPU
- Ubuntu 26.04
- Ryzen 7 9700X
- 64 GB RAM

Server:
- llm-scaler-vllm:0.21.0-b1
- dtype=float16
- kv-cache-dtype=fp8
- max-num-seqs=1
- max-num-batched-tokens=4096
- no async scheduling
- no prefix caching during the controlled test
- Mamba cache dtype left at auto
- XPU Graph disabled during the controlled test

Captured prompt:
- 111,667 exact token IDs
- temperature=0
- max_tokens=1 or streaming
- identical input on every execution

Observed result:

Attempts using the exact same token IDs:

1. correct response
2. correct response
3. correct response
4. first generated characters are `!!!!!!!!`

The failure starts at output position zero, so corruption is already present
after prefill, before decode accumulation.

A separate controlled run with 113,495 exact token IDs produced failures at
iterations 2, 3, 5, 8 and 11 in different test configurations.

## Evidence of non-finite values

Requesting logprobs eventually caused:

`ValueError: Out of range float values are not JSON compliant: nan`

This shows that the corrupted execution can produce NaN logprob values.

Inductor `nan_asserts=true` also raised, although one assertion was triggered
on `z` before the mutating GDN custom op filled that `empty_like` buffer, so
that particular assertion was premature and is not by itself proof of the
root tensor.

## Suspected kernel

The active fallback path calls:

`torch.ops.vllm.gdn_attention_core_xpu(...)`

which dispatches to:

`torch.ops._xpu_C.gdn_attention(...)`

in:

`vllm/_xpu_ops.py::_gdn_attention_core_xpu_impl`

The 0.21.0-b1 source itself contains this comment near the call site:

`the GDN core is the !!!! OOB-corruption region`

Prefill and mixed batches continue using the upstream SYCL GDN kernel.

## Eliminated conditions

The issue still occurs with:

- temperature=0
- max_num_seqs=1
- async scheduling disabled
- prefix caching disabled
- XPU Graph disabled
- packed recurrent decode disabled
- all DISABLE_ESIMD_GDN_* variables enabled
- RMSNormGated XPU kernel replaced temporarily by forward_native
- causal-conv metadata copies made synchronous
- Mamba SSM cache in float32
- both AutoRound checkpoint and online sym_int4 observations

Changing `z = torch.empty_like(core_attn_out)` to `zeros_like` passed a short
20-run test but still failed under real workload, so it is not a complete fix.

## Additional dtype bug

`--mamba-cache-dtype float32` causes deterministic `!!!!!!!!` even for a valid
8,420-token prompt, while Mamba cache dtype `auto` produces coherent output
with the same exact tokens.

## Reproducer artifacts available

- Exact prompt-token JSON for a 111,667-token failure
- Exact prompt-token JSON for a 113,495-token failure
- Streaming reproducer that aborts after 8 consecutive exclamation marks
- Repeat-run JSON results
- Full DEBUG vLLM logs
- Generated Inductor graph and traceback

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.