gpt-oss-120b (MXFP4, TP=4, 4x Arc Pro B70): XPU Graph mode produces garbage ("!!!!" / empty tokens) on both 0.21.0-b3.1 and 0.26.0-b2; eager mode is correct

## Summary

`openai/gpt-oss-120b` runs correctly in eager mode (`--enforce-eager`) on 4x Arc Pro B70 with TP=4 on both `intel/llm-scaler-vllm:latest` (0.21.0-b3.1) and `0.26.0-b2`. As soon as XPU Graph mode is enabled (`VLLM_XPU_ENABLE_XPU_GRAPH=1`, no `--enforce-eager`) the server either fails during graph capture with `UR_RESULT_ERROR_OUT_OF_RESOURCES` (default capture sizes) or, when capture is limited to small batch sizes, starts fine but generates garbage on every request:

- 0.21.0-b3.1: `content` is empty, tokens are produced but decode to nothing (`out=100`, `content=''`, no `reasoning`).
- 0.26.0-b2: classic runaway `!!!!!!!!` degeneration, mixed with random-script tokens (`কঠ কঠ`), from the very first request, also on a 80-token prompt. 27x `Harmony parser ended in a non-terminal state` in ~10 requests.

Decode throughput in graph mode is 100-128 tok/s single-stream (vs. 15-16 tok/s eager), so the graph path would be a ~7x win for this model if the kernels were correct — which is why I'm reporting it rather than just staying on eager.

## Environment

| | |
|---|---|
| GPUs | 4x Intel Arc Pro B70 32 GB (`0xe223`), PCI 03:00.0 / 23:00.0 / c3:00.0 / f3:00.0 |
| Host | AMD Threadripper PRO 9955WX, 125 GB RAM, Ubuntu 24.04.4, kernel **7.1.13** mainline (needed for the multi-root L0 USM fix), Secure Boot off |
| Docker | 29.8.0 |
| Image A | `intel/llm-scaler-vllm:latest` = `0.21.0-b3.1`, digest `032916bd9264…`, vLLM `0.21.1.dev0+gad7125a43.d20260812`, torch `2.11.0+xpu`, vllm-xpu-kernels `0.1.8.3.dev0+g3cab97a.d20260812`, triton-xpu 3.7.0, libze-intel-gpu1 `26.14.37833.4` |
| Image B | `intel/llm-scaler-vllm:0.26.0-b2`, digest `52218ad85513…`, vLLM `0.26.1.dev0+g568afb3a1.d20260907`, torch `2.12.0+xpu` |
| Model | `openai/gpt-oss-120b` (HF, MXFP4 as shipped), `--served-model-name gpt-oss-120b` |

Host RAM note: with 4x B70 the xe driver keeps a pinned host-side mirror of all GPU allocations (4 × 27.8 GB VRAM at `--gpu-memory-utilization 0.80` = 111 GB RAM). All tests below were done at `0.65` (≈20 GB host RAM free, swap empty) to rule out memory starvation — eager mode on both images is verified correct under exactly these conditions immediately before/after each graph run.

## Working baseline (eager) — identical on both images

```bash
docker run -d --name vllm-gptoss \
--privileged --device /dev/dri -v /dev/dri:/dev/dri --ipc=host --shm-size 32g \
-v /opt/hf-cache:/root/.cache/huggingface -v /opt/vllm-cache:/root/.cache/vllm -p 8000:8000 \
-e VLLM_WORKER_MULTIPROC_METHOD=spawn -e ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE -e SYCL_UR_USE_LEVEL_ZERO_V2=0 \
-e CCL_ENABLE_SYCL_KERNELS=0 -e CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0 -e CCL_ATL_TRANSPORT=ofi \
-e CCL_ZE_IPC_EXCHANGE=pidfd -e FI_PROVIDER=shm -e CCL_ATL_SHM=1 -e CCL_WORKER_COUNT=4 \
-e UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 -e ZES_ENABLE_SYSMAN=1 -e TRITON_INTEL_DEVICE_ARCH=bmg \
-e CCL_TOPO_P2P_ACCESS=1 \
intel/llm-scaler-vllm:latest \
openai/gpt-oss-120b --tensor-parallel-size 4 --max-model-len 65536 --enforce-eager \
--gpu-memory-utilization 0.65 --served-model-name gpt-oss-120b --host 0.0.0.0 --port 8000 \
--enable-auto-tool-choice --tool-call-parser openai
```

Result (both images): correct Dutch/English text, correct tool calls (round trip), prompts of 80 / 4k / 15k / 20k tokens fine, 0 Harmony warnings. 15-16 tok/s single stream, ~65 tok/s aggregate at 4 concurrent, KV cache 407k tokens.

## Failing configuration 1: graph mode, default capture sizes → OOM during capture

Same command minus `--enforce-eager`, plus `-e VLLM_XPU_ENABLE_XPU_GRAPH=1`.

- 0.21.0-b3.1 (at 0.70 and 0.80, earlier test): crash-loop, exit 137, `UR_RESULT_ERROR_OUT_OF_RESOURCES` during capture.
- 0.26.0-b2 (at 0.65, 20 GB host RAM free): engine dies after ~175 s:

```
(EngineCore pid=539) RuntimeError: Worker failed with error 'level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)', please check the stack trace above for the root cause
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore': 1}
```

## Failing configuration 2: graph mode, capture limited to small batches → starts, but output is garbage

Same as above plus `--compilation-config '{"cudagraph_capture_sizes":[1,2,4,8]}'`. Server starts (0.21: compile 16.7 s, init 110 s; 0.26-b2: healthy after 120 s), health OK, no errors in the log.

**0.21.0-b3.1** — every request returns tokens that decode to nothing:

```
short prompt= 80 out= 100 content='' reasoning=''
4k prompt= 4274 out= 38 content='' reasoning=''
15k prompt= 15474 out= 9 content='' reasoning=''
tool test: tool_calls: [] finish: length
```

**0.26.0-b2** — runaway `!` degeneration from the first request:

```
short prompt= 80 out= 23 content="'s, stretched ( i.e,1 after each Buh 0.0003!< 3D.<|endoftext|>"
4k prompt= 4274 out= 100 content='- ( this Ton!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
15k prompt= 15474 out= 100 content='\n কঠ কঠ কঠ কঠ কঠ কঠ!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
story prompt= 84 out=1500 content='!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!…'
log: 27x "WARNING [harmony.py:119] Harmony parser ended in a non-terminal state; returning the recovered raw output."
```

Throughput of the (garbage) graph path, 250 output tokens, temperature 0:

| | eager | graph (limited capture) |
|---|---|---|
| 0.21.0-b3.1, 1 stream | 15-16 tok/s | 40-51 tok/s |
| 0.26.0-b2, 1 stream | 13-14.5 tok/s | 100-128 tok/s |
| 0.26.0-b2, 4 concurrent | 57 tok/s | 310-330 tok/s |

Switching the same container back to `--enforce-eager` restores correct output immediately, so model files, tokenizer and the rest of the stack are fine.

## Side observation

After the graph-mode crash (config 1) the next eager container hung at worker init (`parallel_state ... backend=xccl`, 0% VRAM, no further log lines for 16 min) until it was restarted — same symptom as the known "wedged GPU" hang. A plain `docker restart` recovered it; no host reboot needed.

## Ask

1. Is XPU Graph expected to work for gpt-oss (MXFP4 MoE + Harmony) on B70 TP=4 in 0.26? If yes, the `custom_esimd_kernels_vllm / moe_ops` graph path seems to produce wrong logits (the "!" pattern of vllm-project/vllm#45540 / #38994 and issues #647/#680/#695 here, but deterministic and immediate).
2. If graph mode is not supported for this model/quant combination, please make the server refuse `VLLM_XPU_ENABLE_XPU_GRAPH=1` for it instead of serving garbage at 200 OK.
3. Default capture sizes OOM at 0.65-0.80 GPU memory utilization on 32 GB cards; a documented safe `cudagraph_capture_sizes` for 4x B70 would help.

I can rerun any of these configurations with extra logging on request; each swap takes ~3 minutes on this box.