[GSD-13010] Sporadic permanent GPU wedge on dual Arc Pro B70 (BMG G31): ccs/bcs engine reset + "Fault response: Unsuccessful -ENOENT/-EINVAL" under sustained Level-Zero inference load

### Pre-submission Checklist

- [ ] I am using the latest GPU driver version — **currently on distro-provided 26.05.37020.3 (Ubuntu 26.04); latest release is 26.22.38646.4. Happy to retest on the latest release if you expect it to be relevant — the failure is easy for us to reproduce.**
- [x] I have searched for similar issues and found none — closest field reports: vllm-project/vllm#41663 (same dual-B70 hardware, GP fault + bcs engine reset) and darktable-org/darktable#20257 (same ccs engine-reset signature on Arc B580 via OpenCL, no vLLM/oneCCL involved).

# System Information

### GPU Hardware

2× Intel Arc Pro B70 (Battlemage G31, PCI ID `8086:e223`, ASRock subsystem `6025`)

### DRI Devices Information

```
$ ls -ls /dev/dri/*
0 crw-rw----+ 1 root video 226, 0 Jul 4 18:12 /dev/dri/card0
0 crw-rw----+ 1 root video 226, 1 Jul 4 18:12 /dev/dri/card1
0 crw-rw----+ 1 root video 226, 2 Jul 4 18:12 /dev/dri/card2
0 crw-rw----+ 1 root render 226, 128 Jul 4 18:12 /dev/dri/renderD128
0 crw-rw----+ 1 root render 226, 129 Jul 4 18:12 /dev/dri/renderD129

$ ls -la /dev/dri/by-path/
pci-0000:45:00.0-card -> ../card0 (platform framebuffer)
pci-0000:c3:00.0-card -> ../card1
pci-0000:c3:00.0-render -> ../renderD128
pci-0000:c7:00.0-card -> ../card2
pci-0000:c7:00.0-render -> ../renderD129
```

### GPU Detailed Information (lspci output)

```
$ sudo lspci -vvv -k -s c3:00.0 (c7:00.0 is identical)
c3:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device 6025
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Region 0: Memory at 11e00000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at 10800000000 (64-bit, prefetchable) [size=32G]
Kernel driver in use: xe
Kernel modules: xe
```

Host platform: AMD EPYC 7413 (24c). GuC firmware: `xe/bmg_guc_70.bin` version **70.58.0**.

### Driver Version

intel-opencl-icd / libze-intel-gpu1 **26.05.37020.3-1** (Ubuntu 26.04 distro packages)

### Installed GPU Driver Packages

```
intel-opencl-icd 26.05.37020.3-1
intel-opencl-icd-legacy 24.35.30872.45-1
libze-intel-gpu-dev 26.05.37020.3-1
libze-intel-gpu-legacy1-1 24.35.30872.45-1
libze-intel-gpu1 26.05.37020.3-1
libze1:amd64 1.28.2-2
```

### Driver Installation Details

Distro-provided packages from Ubuntu 26.04 LTS archives (no Intel PPA / no manual install). Workload runs inside Docker containers with `--device /dev/dri` passthrough; the failure reproduces with two different container userspaces (oneAPI 2025.3.2 and the intel/vllm 0.17 prebuilt image), which points away from a specific container userspace.

### Linux Distribution

Other

### Other Linux Distribution

Ubuntu 26.04 LTS

### Kernel Version & Boot Parameters

```
$ uname -a
Linux battlematrix 7.0.0-27-generic #27-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 18 19:13:49 UTC 2026 x86_64 GNU/Linux

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.0-27-generic root=UUID=... ro quiet splash crashkernel=2G-4G:320M,...
```

# Issue Description

### Actual Behavior

Under sustained multi-request LLM inference (vLLM, tensor-parallel=2 across the two B70s, 8–12 concurrent requests, mixed 4k–32k contexts), roughly **once every 2–6 hours** one card takes a GPU page fault and the xe driver resets its compute/copy engines:

```
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=29, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
```

After the reset the userspace Level-Zero context is **permanently wedged**: the in-flight SYCL kernel never completes, the worker process never returns, and the serving engine dies (`TimeoutError: RPC call to sample_tokens timed out` → `EngineDeadError`). Only a full process/container restart recovers.

Key observations:
- **Both cards are affected** (events captured on `c3:00.0` and `c7:00.0`) → not a single defective unit.
- 8 application-level failures over July 2–4 **each correlate to the second** with a dmesg engine-reset event.
- The dying scheduler step can be **tiny**: one captured crash dump shows 9 requests in decode, 9 total scheduled tokens, KV-cache at 14% — so this is not a batch-size/pressure problem. Incidence appears to scale with the number of kernel submissions over time, not with step size.
- Ruled out on our side: model & quantization path (reproduces with int4/AWQ/FP8 weights), guided-decoding backend, vLLM graph capture (enforce-eager), vLLM async scheduling (disabled), vLLM version (intel/vllm 0.17 image and upstream v0.24 source build), scheduler oversubscription (max-num-seqs capped).
- oneCCL "stable fallback" env from vllm-project/vllm#41663 already applied in all runs (`CCL_ENABLE_SYCL_KERNELS=0`, `CCL_ATL_TRANSPORT=ofi`, `CCL_ZE_IPC_EXCHANGE=sockets`, `CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0`, `SYCL_UR_USE_LEVEL_ZERO_V2=0`) — it removed an earlier crash mode but not this one.
- Same ccs engine-reset signature has been reported without vLLM/oneCCL entirely (darktable-org/darktable#20257, OpenCL on B580), suggesting driver/firmware level.

### Expected Behavior

No GPU page faults / engine resets under sustained compute load; or, if an engine reset occurs, the Level-Zero context should recover (or at least fail fast with an error the application can handle) instead of wedging permanently.

### Reproduction Rate

Always reproduces - 100% (within 2–6 hours of sustained load; see steps)

### Steps to Reproduce

1. Host as described above (dual Arc Pro B70, xe driver, kernel 7.0.0-27).
2. Serve a ~27B LLM with vLLM v0.24 (XPU build from upstream `Dockerfile.xpu`, oneAPI 2025.3.2) with `--tensor-parallel-size 2 --enforce-eager --max-model-len 32768`, oneCCL env as above. (Also reproduces with the intel/vllm:0.17.0-xpu prebuilt image and a 70B AWQ model.)
3. Drive sustained concurrent load: 8–12 parallel `/v1/chat/completions` requests in a loop for hours (we use JSON-schema constrained outputs, mixed prompt sizes 4k–32k tokens).
4. Within 2–6 hours: dmesg shows the ccs/bcs engine reset + unsuccessful fault response on one of the cards, and the serving engine hangs permanently.

Minimal load generator (python, against the vLLM OpenAI endpoint):

```python
import httpx, json, time, concurrent.futures as cf
SCHEMA = {"type":"object","properties":{"verdict":{"type":"string"},"reasoning":{"type":"string"}},
"required":["verdict","reasoning"]}
def call(i):
p = f"Analyze company #{i}. Respond with JSON matching: {json.dumps(SCHEMA)}"
return httpx.post("http://127.0.0.1:8080/v1/chat/completions", timeout=600, json={
"model":"local","messages":[{"role":"user","content":p}],
"max_tokens":300,"response_format":{"type":"json_object"}}).status_code
while True:
with cf.ThreadPoolExecutor(max_workers=12) as ex:
print(time.strftime("%T"), list(ex.map(call, range(12))))
```

# Regression Information

### Is this a regression?

- [ ] Yes, this is a regression — unknown: this dual-B70 configuration has shown the issue on every stack we tried; we have no known-good baseline.

# Logs

### API Call Logs

Not captured yet — the incident is frequent enough to instrument: we can provide `ze_debug`/API traces, `py-spy`/gdb stacks of the hung worker process, or xe debugfs state on request. Please tell us which traces are most useful.

### strace Logs

Not captured (available on request, see above).

### System Logs / dmesg Output

All engine-reset events captured July 2–4 (timestamps CEST). Every one coincides (±2 s) with an application-level engine death:

```
[Jul 2 23:40:09] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=43, state=0x289
[Jul 2 23:40:09] xe 0000:c3:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 2 23:53:37] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=48, state=0x289
[Jul 3 00:01:23] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=27, state=0x289
[Jul 3 00:01:23] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=32, state=0x289
[Jul 3 00:12:24] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=26, state=0x289
[Jul 3 00:18:56] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=43, state=0x289
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
[Jul 4 05:49:27] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=48, state=0x289
[Jul 4 10:51:14] xe 0000:c3:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=31, state=0x289
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
```

### Backtrace (if crash or hang occurred)

Application-side (vLLM EngineCore parent; the GPU worker itself hangs and never returns — we can attach py-spy/gdb to it at the next occurrence on request):

```
(EngineCore) ERROR [core.py:1233] EngineCore encountered a fatal error.
File ".../vllm/v1/executor/multiproc_executor.py", line 387, in get_response
status, result = mq.dequeue(timeout=dequeue_timeout)
File ".../vllm/v1/executor/multiproc_executor.py", line 370, in get_response
raise TimeoutError(f"RPC call to {method} timed out.") from e
TimeoutError: RPC call to sample_tokens timed out.
(APIServer) vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.
```

Captured scheduler state at one crash (shows the dying step was 9 decode tokens across 9 requests, KV 14%):

```
SchedulerOutput(..., total_num_scheduled_tokens=9, ...)
SchedulerStats(num_running_reqs=9, num_waiting_reqs=0, kv_cache_usage=0.1439...)
```

# Reproducer

### Source Code / Reproducer

See "Steps to Reproduce" above — vLLM serve command plus the 15-line python load generator. Full container logs, complete dmesg, and our original forensics write-up are in #946 (filed before we were aware of the template — apologies; this issue supersedes it).

No repository source file or test is named. Start with the supplied vLLM v0.24 command and 15-line Python load generator on the dual B70 setup, then correlate xe dmesg reset/fault timestamps with the hung worker. Done means identifying the driver, firmware, or Level-Zero failure boundary and establishing whether the permanent wedge can be prevented or made recoverable.

Written by the indexing model from the issue text.