intel / intel/llm-scaler

[Bug] Qwen3-Coder-Next fails on sym_int4: moe_topk_softmax unsupported (E=512, top_k=10) — dual Arc Pro B70, 0.14.0-b8.3.1

Open
#477 1 comment 1 reaction 1 assignee Claimed by @glorysdj View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

## Summary
Qwen3-Coder-Next (Qwen3NextForCausalLM, 512 experts / top-k 10) fails during the
first forward pass (profile_run) with sym_int4 online quantization on
`intel/llm-scaler-vllm:0.14.0-b8.3.1`. Weight loading and quantization complete
successfully (19.8 GiB per card), but the MoE routing kernel rejects the model's
expert configuration:

```
RuntimeError: moe_topk_softmax: unsupported (E=512, top_k=10) — add a moe_topk_v2_host instantiation
```

Since the b8.3 release notes mention performance improvements for Qwen3-Coder-Next,
I assume this model is intended to be supported — it looks like the (E=512, top_k=10)
shape is missing from the precompiled kernel instantiations.

## Environment
- Image: `intel/llm-scaler-vllm:0.14.0-b8.3.1` (vLLM 0.14.1.dev0+gb17039bcc.d20260605)
- GPUs: 2× Intel Arc Pro B70 32GB (BMG-G31, device 0xe223), full 32GB BAR on both
- Host: Ubuntu Server 24.04.4 LTS, HWE kernel 6.17.0-35-generic
- Host platform: i9-9900K / Z390, cards at PCIe 3.0 x8/x8 (kernel params: `xe.max_vfs=0 pci=realloc=off`)
- Container started per README: `--privileged --net=host --device=/dev/dri --shm-size=32g`

## Model
- `Qwen/Qwen3-Coder-Next` (official BF16 checkpoint, 40 shards, verified complete)

## Repro
```bash
export ZE_AFFINITY_MASK=0,1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=0
export CCL_WORKER_COUNT=2
export FI_PROVIDER=shm
export CCL_ATL_TRANSPORT=ofi
export CCL_ZE_IPC_EXCHANGE=sockets
export CCL_ATL_SHM=1
export VLLM_QUANTIZE_Q40_LIB="/usr/local/lib/python3.12/dist-packages/vllm_int4_for_multi_arc.so"

vllm serve /llm/models/Qwen3-Coder-Next \
--served-model-name qwen3-coder-next \
--host 0.0.0.0 --port 8000 \
--tensor-parallel-size 2 \
--quantization sym_int4 \
--dtype float16 \
--enforce-eager --trust-remote-code \
--gpu-memory-utilization 0.85 \
--max-model-len 32768 \
--block-size 64
```

## Observed behavior
- Architecture resolves correctly (`Qwen3NextForCausalLM`), both workers initialize
(TP rank 0/1, EP rank 0/1, xccl backend)
- All 40 shards load and quantize: `Peak XPU memory after loading weights:
peak=28.79 GiB, current=19.80 GiB` on both ranks
- Crash in `profile_run` → `_dummy_run` at the first MoE layer with the
`moe_topk_softmax: unsupported (E=512, top_k=10)` RuntimeError (full log attached)

## Also tried (all fail)
- Without `VLLM_QUANTIZE_Q40_LIB`: same failure
- `VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1`: worker OOM-killed on host (31GB system RAM,
2 workers staging BF16) — unrelated, just noting it was tested
- Pre-quantized GPTQ checkpoint (`btbtyler09/Qwen3-Coder-Next-GPTQ-4bit`) via the
ipex backend: fails differently — `UR_RESULT_ERROR_OUT_OF_RESOURCES` in
`intel_extension_for_pytorch ... marlin_shuffle_weight` during MoE fusion init
(weights 23.34 GiB + ~9 GiB fusion overhead exceeds 32GB). So there is currently
no working INT4 route for this model on 32GB cards.

## Request
Add the (E=512, top_k=10) instantiation for moe_topk_softmax / moe_topk_v2_host so
Qwen3-Coder-Next can run on Arc Pro B-series. Happy to test a candidate build —
this is a dedicated dual-B70 inference box and turnaround on my side is fast.

Full serve log attached: serve-int4.log:

[serve-int4.log](https://github.com/user-attachments/files/28885711/serve-int4.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.