[BMG B65 / xe / GuC 70.72.1] Dual-GPU TP2 workload: GuC consumes SCHED_CONTEXT but leaves enabled LRC undispatched; ring tail advances/head stalls until 'not started' timeout

Proposed primary destination: https://github.com/intel/compute-runtime/issues
(justification in filing-recommendation section of the report.)
Secondary cross-reference: freedesktop GitLab DRM Xe tracker.

---

## Summary

On 2× Intel Arc Pro B65 (Battlemage G31, `8086:e222`) under the xe driver,
a deterministic two-GPU tensor-parallel LLM serving startup (vLLM TP=2,
80 MiB FP16 all-reduce during warmup) fails every time on the default
collective path while two alternative collective algorithms pass every
time on identical hardware and inputs.

Host-side submission is proven complete by kernel trace: job create →
exec → run → submit → H2G CTB `SCHED_CONTEXT` (`0x1000`) write → MMIO
GuC doorbell, all within ~1 ms, with the queue `REGISTERED|ENABLED`.
Live GuC CTB sampling shows the firmware consuming every message
(memory-head tracks tail through the failure, `broken=0`). Yet the LRC
ring head never advances, the engine stays idle, no page fault occurs,
and ~108 s later the job times out as "not started" (`error -62`).

So: a received, enabled GuC context is sometimes never dispatched by GuC
firmware under this reproducible dual-GPU workload. `TMP_BUF`-style
avoidance (a different submission pattern via `CCL_SYCL_ALLREDUCE_TMP_BUF=1`)
never triggers it (5/5 clean), which localizes the trigger to the default
submission sequence, not to data movement or memory capacity.

## Hardware

- 2× Intel Arc Pro B65 32 GB, Battlemage G31, PCI `8086:e222`, subsys `1849:6027`
- `06:00.0` chipset-attached (rank 0), `0c:00.0` CPU-attached (rank 1); both `xe`
- Board: AMD B550, ASRock; host Fedora CoreOS; boot includes `pci=noats`
- P2P verified working both directions (~2.7–3.1 GB/s bulk)

## Software

- Fedora kernel `7.1.8-200.fc44.x86_64`, xe driver (in-tree)
- GuC firmware `xe/bmg_guc_70.bin` **70.72.1**, RUNNING both cards
(from `intel-gpu-firmware-20260810-1.fc44.noarch`; only GuC blob on disk)
- Level Zero userspace `26.27.39122.11-0` (`libze-intel-gpu1`), IGC 2.38.2
- torch `2.13.0+xpu`, oneCCL `Gold-2022.0.0`, SYCL RT 2026.0.0
- vLLM `0.28.1rc1.dev388+g8a728663c.xpu`, vllm-xpu-kernels `0.1.14.1`

## GuC firmware

`bmg_guc_70.bin`, version 70.72.1 (wanted 70.54.0), RUNNING on both cards,
IP ver 20.2.0. Only `bmg_guc_70` blob present in `/lib/firmware/xe/`.

## Reproduction

Model: Qwen3.8-27B OrcaRouter AutoRound W4A16/G128 (public Hugging Face
weights), served with vLLM TP=2, FP16 activations, FP8 KV,
`--max-model-len 262144 --max-num-seqs 32 --max-num-batched-tokens 8192
--gpu-memory-utilization 0.95`, PIECEWISE XPU graphs. Exact launch command
in `reproduction.txt`.

Expected: engine passes `determine_available_memory`/profile warmup.
Actual: during warmup, right after an 80 MiB `[8192,5120]` FP16 all-reduce
is dispatched on both ranks, both workers fail at the next synchronize
with `UR_RESULT_ERROR_DEVICE_LOST` / `UR_RESULT_ERROR_OUT_OF_RESOURCES`;
dmesg shows GuC "not started" timeouts + engine resets on both GPUs.

Note: a standalone model-free 80 MiB all-reduce passes — full
model-resident/profile state (weights resident, compiled profiling) is
required for deterministic reproduction.

Reproduction rate: default 5/5 fail; `CCL_SYCL_ALLREDUCE_TMP_BUF=1` 0/5
fail (profile completes); `CCL_ALLREDUCE=ring` 0/5 fail. Single-GPU and
dual-independent-replica serving of the same model work.

## Failure signature

- Timed-out job "not started" (e.g. `0c:00.0`, guc22, seqno 360), no page
faults required, predecessor completes normally, DRM dependencies drain,
no TLB timeouts, no earlier stuck occupant in 15K-line kernel trace.
- Preserved devcoredumps (separate equivalent failures): single pending
context, LRC tail advanced (~1120 B and ~2352 B cases), last completed
seqno = timedout − 1, engine registers idle.

## Evidence chain

1. xe schedules/submits job (create/exec/run/submit trace records, 1:1).
2. Ring tail advances (coredump LRC state).
3. H2G `SCHED_CONTEXT` emitted (trace record, action `0x1000`).
4. GuC notify/MMIO doorbell invoked (kprobe record, same ms).
5. GuC CTB head consumes the submission (1 Hz sampled head==tail,
advancing, `broken=0`).
6. LRC head never advances; engine idle.
7. Context remains undispatched until "not started" timeout + reset.

## Control

`CCL_SYCL_ALLREDUCE_TMP_BUF=1` avoids the issue (different submission/
memory pattern, much faster than forced ring). This is a workaround
observation, not a claim that oneCCL is defective — the failure as
localized is below the L0 API, in GuC consumption-to-scheduling.

## Expected / actual

Expected: GuC schedules the enabled context after consuming its
SCHED_CONTEXT. Actual: GuC consumes control traffic but the context sits
pending/idle until TDR.

## Attachments

`summary.json` (machine-readable findings), `dmesg-key-window.txt`
(timeout + reset block), `xe-trace-key-window.txt` (submit→silence→
timeout for the stuck job), `guc-ctb-key-series.csv` (head==tail through
failure), `devcoredump` (+metadata showing pending context/idle
engine), `version-inventory.txt`, `reproduction.txt` (exact launch),
`SHA256SUMS`. Full raw archives (trace, server logs, 1 Hz CTB series)
available on request.

## Questions for Intel

1. Does this match a known GuC scheduling issue on BMG / GuC 70.72.1?
2. Are there newer or older GuC firmware versions you recommend A/B
testing (only `bmg_guc_70` is on disk here)?
3. Is there a GuC firmware log decoder/tool available for the preserved
level-1 `guc_log` dumps?
4. Would higher GuC log verbosity help, and what exact module/debugfs
configuration should be used?
5. Is there additional firmware/scheduler state in the devcoredump you
want captured from the next failure?
6. Is `CCL_SYCL_ALLREDUCE_TMP_BUF=1` an acceptable temporary workaround?

## Related issues (reviewed; filing separately rather than attaching)

- intel/compute-runtime #948 (dual-B70 sporadic page-fault wedge, GuC 70.58.0): same die class and TP=2 shape, but sporadic multi-hour fault-driven wedge vs this deterministic zero-fault startup hang on newer firmware. Probably distinct mechanism; shared GuC-scheduling neighborhood.
- uxlfoundation/oneCCL #212 (stale IPC handle): mechanism-adjacent page-fault path, but this failure reproduces with IPC caches disabled and zero faults, and GET reuse is structurally bypassed on the failing path.
- uxlfoundation/oneCCL #215 and vllm-project/vllm#50545 (B60 large-AR warmup hang): same symptom family, closed with workarounds on older stacks; this report adds scheduling-level localization (submit→doorbell→CTB-consumed, never dispatched) those threads lack.
- intel/compute-runtime #952/#953 (host-RAM pin OOM): considered and excluded (different mechanism).

## Key evidence excerpts

dmesg: GuC timeout + reset block (both GPUs)

```
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: Check job timeout: seqno=2428257, lrc_seqno=2428257, guc_id=0, not started
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: Timedout job: seqno=2428257, lrc_seqno=2428257, guc_id=0, flags=0x73 in no process [-1]
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: trying reset from guc_exec_queue_timedout_job [xe]
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: reset queued
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: reset started
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: reset done
[Thu Sep 17 19:21:41 2026] xe 0000:06:00.0: [drm] Tile0: GT0: Timedout job: seqno=330, lrc_seqno=330, guc_id=22, flags=0x20 in python [924129]
```

xe trace: stuck job submit → silence → timeout (0c/guc22, seqno 360)

```
VLLM::Worker_TP-901818 [008] ..... 690903.176768: xe_sched_job_exec: dev=0000:0c:00.0, fence=00000000e8abfb21, seqno=359, lrc_seqno=359, gt=0, guc_id=22, batch_addr=0xffffd556a77f1ac0, guc_state=0x3, flags=0x20, error=0
kworker/u48:2-902171 [011] ..... 690903.176774: xe_sched_job_run: dev=0000:0c:00.0, fence=00000000e8abfb21, seqno=359, lrc_seqno=359, gt=0, guc_id=22, batch_addr=0xffffd556a77f1ac0, guc_state=0x3, flags=0x20, error=0
VLLM::Worker_TP-901818 [008] ..... 690903.259596: xe_sched_job_exec: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x3, flags=0x20, error=0
kworker/u48:4-888717 [005] ..... 690903.259605: xe_sched_job_run: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x3, flags=0x20, error=0
kworker/u48:2-902171 [009] ..... 691011.959433: xe_sched_job_timedout: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x200, flags=0x20, error=0
kworker/u48:2-902171 [009] ..... 691011.959434: xe_sched_job_set_error: dev=0000:0c:00.0, fence=00000000aa8e88b5, seqno=360, lrc_seqno=360, gt=0, guc_id=22, batch_addr=0xffffd556a77f1b80, guc_state=0x200, flags=0x20, error=-62
```

H2G CTB head==tail through failure (DW units, 1 Hz samples)

```csv
wall,gpu,h2g_mem_head_dw,h2g_mem_tail_dw
2026-09-17T19:21:20,06:00.0,984,984
2026-09-17T19:21:23,06:00.0,990,990
2026-09-17T19:21:26,06:00.0,993,993
2026-09-17T19:21:29,06:00.0,999,999
2026-09-17T19:21:33,06:00.0,1002,1002
2026-09-17T19:21:36,06:00.0,982,982
2026-09-17T19:21:39,06:00.0,982,982
2026-09-17T19:21:42,06:00.0,982,982
2026-09-17T19:21:45,06:00.0,985,985
2026-09-17T19:21:49,06:00.0,484,908
```

devcoredump ring state (independent equivalent failure)

Single pending bcs context per GPU (1120 B and 2352 B cases): LRC Head behind Tail, last completed seqno = timedout seqno − 1, Schedule State 0x201, engine registers idle. Full dumps available on request.

Start with reproduction.txt and the attached xe trace, dmesg window, CTB series, and devcoredump to reproduce the default and workaround outcomes. Compare the submission, SCHED_CONTEXT consumption, LRC state, and timeout evidence; done means identifying the driver or firmware cause, or the additional capture needed to isolate it, and validating any proposed change against both paths.

Written by the indexing model from the issue text.