LLM 0.21.0-b2 crashes with `UR_RESULT_ERROR_DEVICE_LOST` during startup memory-profiling on 2x Arc Pro B70 (TP=2) — regression from 0.21.0-b1

# vLLM 0.21.0-b2 crashes with `UR_RESULT_ERROR_DEVICE_LOST` during startup memory-profiling on 2x Arc Pro B70 (TP=2) — regression from 0.21.0-b1

## Environment

- **GPUs:** 2x Intel Arc Pro B70 32GB (Battlemage), PCIe-connected (WRX80E platform), **no XeLink**
- **CPU / platform:** AMD Ryzen Threadripper PRO 3995WX (64-core)
- **OS:** Debian GNU/Linux 13 (trixie), kernel `7.1.3+deb13-amd64`
- **Kernel cmdline (relevant flags):** `amd_iommu=on iommu=pt pci=nommconf pcie_aspm=off pci=pcie_bus_perf`
- **Docker:** `29.6.2`, Docker Compose `v5.3.1`
- **Image (repro 1):** `intel/llm-scaler-vllm:0.21.0-b2`, stock, no modifications.
Digest: `sha256:3f0a8c60fbaf376ec09538f093cba91f171238b99c117445c0bcc6096272ec3e`
- **Image (repro 2):** the same b2 base plus a driver-only overlay (adds
`ppa:kobuk-team/intel-graphics` → `libze1 libze-dev intel-level-zero-gpu intel-opencl-icd
libigc2 libigdfcl2`, no other changes) — built to work around an unrelated
`urContextCreate` multi-device issue seen on the b1 stock driver. **Crashes identically**, so
the driver version is not the variable here.
- **vLLM version (from startup log):** `0.21.1.dev0+gad7125a43.d20260802`
- **Model:** `Qwen/Qwen3.6-35B-A3B` (`Qwen3_5MoeForConditionalGeneration` — hybrid MoE +
GDN/mamba architecture), served as `qwen3.6-35b`
- **Serve config (`vllm serve` args, from the logged `non-default args`):**
```
--tensor-parallel-size 2 --dtype float16 --quantization fp8 --max-model-len 262144
--kv-cache-dtype auto --gpu-memory-utilization 0.9 --max-num-seqs 2
--max-num-batched-tokens 16384 --block-size 64 --language-model-only
--enable-auto-tool-choice --tool-call-parser qwen3_coder --trust-remote-code
```
- **Relevant env vars:** `VLLM_TARGET_DEVICE=xpu`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`,
`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`, `VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1`,
`ZE_AFFINITY_MASK=0,1`, `ONEAPI_DEVICE_SELECTOR=level_zero:0,1`,
`VLLM_XPU_ENABLE_XPU_GRAPH=0` (XPU graph capture explicitly disabled — unrelated to this
crash, see below)

## Summary

`intel/llm-scaler-vllm:0.21.0-b2` crashes 100% of the time (3/3 configurations tested) during
vLLM's own internal startup memory-profiling pass, at the TP=2 collective, with:

```
RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
```

and a corresponding real GPU hang/reset visible in `dmesg` at the exact same timestamp. The
identical model, identical serve config, identical hardware runs correctly on
`intel/llm-scaler-vllm:0.21.0-b1` (or the b1-based driver-overlay build) right now — **this is a
regression between b1 and b2**, not a hardware, driver, or model-config problem.

## Reproduction steps

1. Pull `intel/llm-scaler-vllm:0.21.0-b2`.
2. Serve `Qwen/Qwen3.6-35B-A3B` with the exact flags listed above, `--tensor-parallel-size 2`,
on 2x Arc Pro B70 (PCIe, no XeLink).
3. Observe: model loads and weights load successfully (confirmed both with default
torch.compile and with `--enforce-eager`); the crash happens ~50–100s later, during the
engine's own `determine_available_memory()` dummy-run pass, before any user request is ever
served.

## Observed behavior — Test A: default (torch.compile) mode

Weights load (~107s), torch.compile succeeds (`Compiling a graph for compile range (1, 16384)
takes 53.42s`, `torch.compile took 70.98s in total`). Then:

```
File ".../vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
self.model_runner.profile_run()
File ".../vllm/v1/worker/gpu_model_runner.py", line 5949, in profile_run
hidden_states, last_hidden_states = self._dummy_run(...)
...
File ".../inductor_cache/xz/cxzv2tua2fxvbsmpfgdmyvhheqz27o2b3ui2gcwnkd4zwliixcwo.py", line 1966, in call
torch.ops._c10d_functional.all_reduce_.default(buf3, 'sum', '3')
RuntimeError: level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)
```

Worker cleanup then also fails trying to synchronize the now-dead device:
```
File ".../vllm/v1/worker/gpu_model_runner.py", line ..., in _cleanup_profiling_kv_cache
torch.accelerator.synchronize()
RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
```

`EngineCore failed to start`. **The container does not exit cleanly** — both TP worker
processes hang spinning at 300%+ CPU indefinitely; the health check never passes; Docker's stop
grace period elapses and the container has to be SIGKILLed (`Exited (137)`).

## Observed behavior — Test B: `--enforce-eager` (rules out torch.compile as the cause)

Added `--enforce-eager`. Weights load fine. ~54s later, same internal call fails — but with
**no compiled graph involved at all**:

```
File ".../vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
self.model_runner.profile_run()
File ".../vllm/v1/worker/gpu_model_runner.py", line 5959, in profile_run
self._sync_device()
File ".../vllm/v1/worker/gpu_model_runner.py", line 1065, in _sync_device
torch.accelerator.synchronize()
File ".../torch/accelerator/__init__.py", line 263, in synchronize
torch._C._accelerator_synchronizeDevice(device_index)
RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
```

Same hang-not-exit behavior. This rules out torch.compile/inductor's AOT-compiled all-reduce as
the cause — the crash occurs at the identical internal call regardless of compiled vs. eager
execution.

## Observed behavior — Test C: Kobuk driver-overlay build (rules out stock driver as the cause)

Same config as Test B, only the image swapped to the driver-overlay build described above
(newer `libze1`/`intel-level-zero-gpu`/`libigc2` via `ppa:kobuk-team/intel-graphics`). **Identical
crash**, same call site, same error, same hang. Rules out "stock driver too old for multi-GPU"
as the explanation.

## dmesg evidence (real hardware-level hang, not just a software resource cap)

`sudo dmesg -T | grep -iE 'xe |timedout|reset'` around the Test A crash:

```
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Tile0: GT0: Check job timeout: seqno=1484958, lrc_seqno=1484958, guc_id=0, not started
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Tile0: GT0: Timedout job: seqno=1484958, lrc_seqno=1484958, guc_id=0, flags=0x73 in no process [-1]
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Xe device coredump has been created
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Tile0: GT0: trying reset from guc_exec_queue_timedout_job [xe]
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Tile0: GT0: reset queued
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Tile0: GT0: reset started
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Tile0: GT0: reset done
[Tue Aug 4 20:51:21 2026] xe 0000:2b:00.0: [drm] Tile0: GT0: Timedout job: seqno=325, lrc_seqno=325, guc_id=17, flags=0x20 in python3 [2951393]
```

Local time `20:51:21` matches the crash's logged UTC timestamp (`10:51:21`/`10:51:28`) almost to
the second. `xe` genuinely detected a hung GPU job and reset the tile — this is not merely a
soft resource-accounting error.

After the crash, `sudo xpu-smi discovery` and `sudo xpu-smi dump -d 0,1 -m 0,18,22 -n 1` confirm
both GPUs recovered cleanly: both enumerate normally, both report ~0% compute utilization and
near-zero memory (26.8 MiB / 35.2 MiB — idle baseline, nothing leaked). No lasting GPU damage.

## Regression check: b1 works fine on the identical host

Switched only the image back to the b1-based build (`0.21.1.dev0+gad7125a43.d20260709`),
otherwise byte-identical serve config, same two GPUs (already recovered from the b2 crashes
above), no host changes. **Result: `Application startup complete`, healthy, serves real
completions correctly:**

```json
{"choices":[{"text":" Paris, a city renowned for its iconic landmarks such","finish_reason":"length"}],
"system_fingerprint":"vllm-0.21.1.dev0+gad7125a43.d20260709-tp2-cdb97ae8"}
```

This isolates the crash to the b2 build itself — same host, same hardware, same model config,
b1 works and b2 doesn't.

## What this rules out

- Not a torch.compile/inductor-specific bug (Test B, eager mode, crashes identically)
- Not the stock-vs-newer driver (Test C, newer driver crashes identically)
- Not a host/kernel/hardware regression (Test D, b1 works fine on the same host/GPUs right
after the b2 crashes)
- Not `VLLM_XPU_ENABLE_XPU_GRAPH` (already disabled the entire time; XPU graph capture/replay
is not involved — this crash is inside vLLM's own internal profiling pass, before any graph
capture would occur)

## Possibly related

[#594](https://github.com/intel/llm-scaler/issues/594) reports a different but related-looking
failure on `vllm-0.21.0-b2` with 2x B60 + `Qwen/Qwen3.6-27B`:
`RuntimeError: Worker failed with error 'oneCCL: ze_call.cpp:28 do_call: EXCEPTION: ze error at'`
(`zeMemOpenIpcHandle` → `ZE_RESULT_ERROR_INVALID_ARGUMENT`). Different error string and
hardware, but the same general shape — a multi-GPU TP collective/IPC path failing in b2 — which
may point to a common regression in b2's oneCCL/collective plumbing across the B-series lineup,
not something specific to B70 or this particular model.

## Logs

[README.md](https://github.com/user-attachments/files/30702522/README.md)
[test-c-kobuk-overlay-b2-crash-EXCERPT-ONLY.log](https://github.com/user-attachments/files/30702527/test-c-kobuk-overlay-b2-crash-EXCERPT-ONLY.log)
[xpu-smi-post-crash-health.txt](https://github.com/user-attachments/files/30702525/xpu-smi-post-crash-health.txt)
[dmesg-xe-timeouts.txt](https://github.com/user-attachments/files/30702523/dmesg-xe-timeouts.txt)
[test-d-b1-healthy-startup.log](https://github.com/user-attachments/files/30702526/test-d-b1-healthy-startup.log)
[test-b-stock-b2-eager-crash.log](https://github.com/user-attachments/files/30702524/test-b-stock-b2-eager-crash.log)
[test-a-stock-b2-compiled-crash.log](https://github.com/user-attachments/files/30702528/test-a-stock-b2-compiled-crash.log)