torch.compile silently corrupts ~0.2-0.3% of generations on 2x Arc Pro B70 (TP=1, DP, no P2P) — not graph replay, not TP; the `(a!)` schema fix was built and does NOT resolve it

Filed as a new issue rather than a comment because this is a different configuration from the
existing reports and carries two tested negatives that are easy to lose in a thread.

**Related:** #603 (same `!`-degeneration symptom, 2x Arc Pro B50, TP=2) and #595 — our earlier
report from this same B70 host, filed under our other account @xitation (startup
`UR_RESULT_ERROR_DEVICE_LOST`, resolved by @Wesley-Du's CCL workaround).

**The most useful thing here for #603:** that thread's two reporters both run **TP=2 with the
no-P2P workaround**, so TP/P2P is the natural suspect. **We reproduce the identical symptom at
TP=1**, data-parallel, with no collective communication involved at all. The bug is not in the
TP path. There is also a **working configuration** at the bottom for anyone currently unable
to get non-`!` output.

## Summary

On `intel/llm-scaler-vllm:0.21.0-b3.1` and `:0.21.0-b1`, Qwen3.6-35B-A3B with `sym_int4`,
any **compiled** configuration silently emits degenerate output (a long run of `!`, which is
token id 0) at **0.15%-0.32% of generations** under sustained load, depending on the card and
run (see the per-card caveat below). `--enforce-eager` is clean over **7,296** generations,
and clean in every eager configuration we have run — 0 collapses in over 20,000 generations.

The corruption tracks **compilation**, not graph capture/replay: with `VLLM_XPU_ENABLE_XPU_GRAPH`
explicitly off and compile still on, it persists at the same rate.

`esimd_fake_ops.py` in your own image warns about a mechanism that would produce exactly this
failure. We took that warning seriously, implemented the fix it prescribes, and it did not
resolve the corruption -- see "We tested the obvious fix". Its header:

> The C++ schemas declare those buffers as plain `Tensor` (no `(a!)` mutation annotation)…
> Under torch.compile functionalization, an op declared side-effect-free whose result is
> unused is a candidate for dead-code elimination — which would leave the output buffers
> uninitialised (garbage / empty output)… the proper fix is to re-declare the schemas with
> `(a!)` mutation annotations (requires a C++ rebuild)… Until then, VERIFY output end-to-end
> — do not assume graph mode is correct just because it runs.

We are reporting three separate things: (1) a reproducible silent-corruption rate in compiled
mode, cause still open; (2) the missing-`(a!)` schema defect, which is real and worth fixing
regardless; and (3) our measurement showing (2) does **not** explain (1).

## A collapse is invisible to normal monitoring

Full speed, HTTP 200, `finish_reason: length`. Degenerate output is *cheaper* to generate
than real text, so throughput looks **better**. Nothing short of sampling the generated text
detects it. We only found it because we score every response.

## Measured

gpu-host-02, 2× Arc Pro B70 (`8086:e223`), `xe`, host compute-runtime 26.27 (also 26.05).
Data-parallel 2× TP=1, `--dtype float16 --quantization sym_int4 --max-model-len 32768`,
`--gpu-memory-util 0.88`. Sustained back-to-back batches at TC=48, `max_tokens=512`, every
response scored for longest repeated-character run. **~190,000 correctness-gated generations
across the campaign**, of which >29,000 are eager and contain **zero** collapses.

| config | tok/s | collapses | rate |
|---|---|---|---|
| **b3.1 `--enforce-eager`** | 1173 | **0 / 7,296** | 0 |
| b3.1 `--enforce-eager` @TC=96 | **1348** | **0 / 3,648** | 0 |
| b1 `--enforce-eager` | 822 | **0 / 5,376** | 0 |
| b1 graph | 1602 | 26 / 10,560 | 0.246% |
| b3.1 graph | 1534 | 22 / 10,176 | 0.216% |
| b1 graph (rt 26.05) | 1542 | 162 / 61,248 | 0.264% |
| **compile-only, XPU graph OFF** | 476 | 7 / 3,264 | **0.214%** |
| graph + ESIMD ops in `splitting_ops` | 1531 | 15 / 10,176 | 0.147% (p≈0.25, n.s.) |
| graph + `combo_kernels:false` | 1526 | 20 / 10,176 | 0.197% |

The compile-only row is the important one: **graph replay is not required to reproduce this.**

Collapses **cluster** — bursts separated by clean stretches, not a constant per-request
probability. Consistent with state that accumulates and resets.

## The `(a!)` schema defect (real, but NOT the cause -- see next section)

In `vllm/custom-esimd-kernels-vllm/csrc/xpu/torch_extension*.cc`, ops that receive a
caller-allocated output buffer, write into it, and return it, declare that parameter as a
plain `Tensor`. In principle functionalization is then free to treat the call as pure and DCE
it when the returned value is unused, leaving the buffer uninitialised. We initially believed
this explained the corruption. **It does not** -- we built the fix and measured no significant
change. We report it here because it is still a latent correctness hazard worth closing.

**Both int4 decode-path ops are affected:**

```
esimd_gemv_int4 (Tensor input, Tensor weight, Tensor weight_scale, Tensor output) -> Tensor
esimd_gemm_int4_pgrp(Tensor input, Tensor weight, Tensor weight_scale, Tensor output) -> Tensor
```

Full list of ops with at least one unannotated output buffer (34):

```
esimd_gemv_fp8_pern, esimd_gemv_fp8_pern_fused2, esimd_gemv_fp8_pern_fused3,
esimd_gemv_fp16, esimd_gemv_fp16_gelu_mul, esimd_gemv_fp8_pert,
esimd_gemv_fp8_pert_fused2, esimd_gemv_fp8_pert_fused3, esimd_gemv_int4,
esimd_gemv_int4_fused2, esimd_resadd_norm_gemv_fp8_pert,
esimd_resadd_norm_gemv2_fp8_pert, esimd_norm_gemv_fp8_pert,
esimd_resadd_norm_gemv_int4_pert, esimd_norm_gemv_int4_pert, esimd_rms_norm,
esimd_fused_add_rms_norm, esimd_fused_scaled_add_rms_norm, esimd_rms_norm_gated,
esimd_fused_add_rms_norm_batched, esimd_scaled_resadd_norm_gemv_fp8_pert,
esimd_norm_add_norm, esimd_accum_norm_add_norm, esimd_gemv_fp8_pert_bmg,
esimd_gemm_fp8_pert, esimd_gemm_int4_pgrp, esimd_gemm_fp8_blockscale,
esimd_moe_scatter, esimd_moe_scatter_fused, esimd_moe_gelu_tanh_mul,
esimd_moe_silu_mul, esimd_moe_gather, esimd_moe_gemm_fp8, esimd_moe_gemm_fp8_pert
```

**This is not an unknown pattern in the codebase — 11 ops annotate correctly:**

```
esimd_qkv_split_norm_rope, esimd_qkv_split_norm_rope_v, esimd_qkv_split_norm_rope_onyx,
esimd_gemv_fp8_blockscale_fused2, esimd_gemv_fp8_blockscale_fp16_fused2,
esimd_norm_gemv_fp8_blockscale, esimd_gdn_conv_fused, esimd_gdn_conv_fused_seq,
esimd_gdn_conv_fused_seq_spec, esimd_moe_gemm_fp8_blockscale, q4_0_quantize
```

`esimd_gemm_int4_pgrp` sits in the **same file, ~10 lines above**
`esimd_gemv_fp8_blockscale_fused2`, which correctly declares
`Tensor(a!) o0 … -> Tensor(a!)`. The convention exists and was applied selectively.

## We tested the obvious fix, and it FAILED

We rebuilt all 34 schemas with `(a!)` from your public source and installed the resulting
`.so` files over the shipped ones. Matched pair, same physical card, same day, same load
(card 0, cc=24, graph, `sym_int4`):

| build | collapses | rate | throughput |
|---|---|---|---|
| stock b3.1 | 42 / 12,960 | 0.324% | 760 tok/s |
| **all 34 schemas rebuilt with `(a!)`** | 32 / 13,056 | **0.245%** | 762 tok/s |

**z = 1.20 — not statistically significant.** Throughput unchanged.

"The patch did not apply" is excluded by construction: our image build fails unless
`esimd_gemv_int4` and `esimd_gemm_int4_pgrp` both carry `(a!)` in the *installed* binary, and
that assertion passed. Verified by `strings` on the shipped vs rebuilt `.so`:

```
shipped: esimd_gemm_int4_pgrp(... Tensor output) -> Tensor
rebuilt: esimd_gemm_int4_pgrp(... Tensor(a!) output) -> Tensor(a!)
```

**Conclusion: the missing annotations are a real defect but are not the cause of this
corruption.** The cause is elsewhere in the torch.compile / Dynamo / Inductor path on XPU.
We would rather hand you a tested negative than a plausible story.

For reproduction, the toolchain must link `libsycl.so.8` (matching the shipped kernels):
`intel-oneapi-compiler-dpcpp-cpp` 2025.3.x. The 2026.x compiler links `libsycl.so.9` and
fails the same way `vllm-xpu-kernels` 0.1.13.1 does. Build with `TORCH_XPU_ARCH_LIST=bmg` --
the default target list includes `mtl-h`, where `ocloc` segfaults (exit 245).

## Suggested fix

Re-declare the 34 schemas with `(a!)` (and `(b!)`, `(c!)` for multi-output ops) on the output
parameters and return type, matching the 11 already-correct ops. This will not fix the
corruption reported here -- we measured that -- but it closes a real hazard. Happy to open a
PR; the change is mechanical and we have it working.

**The corruption itself remains open.** We have eliminated the layers listed below; the
remaining surface is the compile stack itself.

## What we falsified (please don't re-run these)

| hypothesis | result |
|---|---|
| cumulative **graph replay** / #507 static buffers | **disproved** — compile-only, replay off, same 0.214% |
| host compute-runtime 26.05 → 26.27 | no-op; the container ships its own UMD (26.14) |
| image regression b1 vs b3.1 | both collapse; b1 is not a safe pin |
| `--max-num-seqs 32` to bound batch size | **5× worse** |
| `cudagraph_capture_sizes` → 128 | OOM (2.67 GiB capture) |
| `DISABLE_ESIMD_INT4` | 3× slower |
| TP=2 | slower than TP=1 — and **TP=1 corrupts too**, so this is not the TP path (relevant to #603) |
| fp32 SSM cache (#559) | worse — 99% `!` on first generation |
| `vllm-xpu-kernels` 0.1.13.1 wheel | `ImportError: libsycl.so.9` on both images |
| `custom_ops:['all']` under compile | `AssertionError: Cannot construct ConstantVariable for torch._C._XpuDeviceProperties` |
| compile mode 1 / `backend=eager` | impossible — no fake impl for `esimd_gemm_int4_pgrp` |

## Three things we did NOT prove

1. **We never observed DCE firing on a specific op, and our evidence now runs against it.**
The schema defect is confirmed by source inspection, but we never captured a compiled
graph with an ESIMD call eliminated — and when we added the annotations and measured, the
corruption did not go away (section above). Separately, forcing the 20 fake-registered
ESIMD ops out via `splitting_ops` moved the rate 0.216% → 0.147%, **not significant**
(p≈0.25); note the int4 decode-path ops have no fakes at all, so they were never traced
and `splitting_ops` could not have addressed them — that arm was near-vacuous by
construction. Treat DCE-of-unannotated-ops as a tested and unsupported hypothesis.
2. **These rates carry much wider error bars than the point estimates suggest, and a
data-parallel aggregate is not comparable to a single-card measurement.** We nearly
reported a false 3x improvement from the `(a!)` patch (z=3.03) that came entirely from
comparing a DP aggregate (which averages both GPUs) against one GPU alone.

We then tested whether our two B70s genuinely differ. Sequential runs suggested yes
(0.324% vs 0.185%, z=2.22, p=0.027); a simultaneous re-run did **not** reproduce it
(0.264% vs 0.179%, z=1.59, p=0.111). Pooled it reaches p=0.007, but pooling was not our
pre-registered decision rule, so **we withdraw the claim** and report it as unresolved.
The practical lesson stands regardless: at ~0.2%, ~13k requests is underpowered to
resolve a 1.5x difference, so measure per-card, pre-register the comparison, and do not
read a single run's p-value as settled.
3. **Greedy decoding is not reproducible across instances on this stack.** Two identically
configured servers agreed on only 9 of 20 `temperature=0` prompts. Worth knowing if you
use cross-instance output comparison in your own validation — it will produce false
positives.

## Also worth flagging: speculative decoding

`platforms/xpu.py` has no gate against speculative decoding, and
`--speculative-config '{"method":"ngram",…}'` runs. Under load it is badly counterproductive
and unstable: acceptance falls from 38.7% at batch=1 to **9.7%** at TC=48, throughput drops
from ~672 to **98.8 tok/s** per card, and the engine then died with
`level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)`. The card recovered.
Consider gating it on XPU until the rejection path is validated.

## Reproducer

Score the longest run of any repeated character in each completion (threshold 80) across
sustained back-to-back batches — a burst will not find it. Our harness (`soak.py`,
`degen_probe.py`, `gpuguard.sh`) is available if useful.

**Note on measurement method:** 3,072 burst-scoped generations across 12 configurations
reported "clean"; 576 *continuous* ones found 0.69%. A burst cannot accumulate the state this
depends on. We nearly shipped a clean bill of health on that basis.

## Working configuration (for anyone stuck on `!` output)

`--enforce-eager` is the only clean configuration we have found. It costs throughput versus
compiled mode but produces correct output:

```
intel/llm-scaler-vllm:0.21.0-b3.1
vllm serve --dtype float16 \
--quantization sym_int4 --allow-deprecated-quantization \
--kv-cache-dtype auto --trust-remote-code \
--gpu-memory-util 0.88 --max-model-len 32768 \
-tp 1 --enforce-eager
```

Run one instance per GPU (data-parallel via `ZE_AFFINITY_MASK`) rather than `-tp 2`; on our
hardware that is both faster and avoids the no-P2P collectives entirely. Measured on 2x B70:
**1348 tok/s aggregate at 96 concurrent requests, 0 collapses in 3,648 generations.** On a
2x B60 host running the same config: 648 tok/s, 0 collapses in 1,824.

Throughput saturates at ~96 concurrent (1077 / 1348 / 1349 tok/s at 48 / 96 / 192), so there
is no benefit to pushing concurrency higher.

**Do not** try to bound the problem with `--max-num-seqs` — we measured that as **5x worse**,
not better.