[Bug] Silent output corruption with --tensor-parallel-size 2 on dual Arc Pro B70: Gemma 4 26B-A4B, FP8, Triton MoE backend (not an alignment issue)

## Summary

Serving `google/gemma-4-26B-A4B-it` with `--quantization fp8
--tensor-parallel-size 2` across two Intel Arc Pro B70 cards produces
**silently incorrect tokens**. The server starts, reports healthy, raises no
exception, and emits fluent grammatical text — but individual tokens are
wrong, deterministically, at temperature 0.

The same model, same image, same quantization is **correct** at
`--tensor-parallel-size 1` and **correct** under `--pipeline-parallel-size 2`.

I want to flag up front that this is **not** the 128-alignment problem that
is usually invoked for this model. I tested that hypothesis directly and it
is falsified — see "What this is not", below.

## Environment

| | |
|---|---|
| Image | `intel/llm-scaler-vllm:0.21.0-b1` @ `sha256:5d87be271e4db54539f1dbb29c071e9122f4e57b74594dbb26a55d27a569d780` |
| vLLM self-report | `0.21.1.dev0+gad7125a43` |
| GPUs | 2 × Intel Arc Pro B70 32 GB (Battlemage), `0000:03:00.0` + `0000:08:00.0`, separate PCIe root ports |
| Host | Ubuntu 26.04, `xe` driver |
| Model | `google/gemma-4-26B-A4B-it` (MoE, 128 experts, `moe_intermediate_size=704`, `intermediate_size=2112`, 30 layers) |
| MoE backend selected | `TRITON` (logged; see below) |

Serve args:

```
vllm serve /llm/models/gemma-4-26B-A4B-it \
--tensor-parallel-size 2 --dtype float16 --quantization fp8 \
--max-model-len 32768 --gpu-memory-utilization 0.90 \
--max-num-batched-tokens 8192 --block-size 64 \
--enforce-eager --trust-remote-code
```

Env: `VLLM_WORKER_MULTIPROC_METHOD=spawn`,
`VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1`, `ZE_AFFINITY_MASK=0,1`,
`OMP_NUM_THREADS=1`, `CCL_TOPO_P2P_ACCESS=0`, `CCL_ATL_TRANSPORT=ofi`,
`CCL_ATL_SHM=1`, `FI_PROVIDER=shm`, `CCL_WORKER_COUNT=2`.

Backend selection line (identical at TP=1 and TP=2):

```
INFO [fp8.py:407] Using TRITON Fp8 MoE backend out of potential backends:
['XPU', 'AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM',
'TRITON', 'MARLIN', 'BATCHED_DEEPGEMM', 'BATCHED_TRITON', 'CPU']
```

## Reproducer

Any deterministic code-generation prompt exposes it. Minimal case:

```
POST /v1/chat/completions
{"model": "...", "temperature": 0, "max_tokens": 200,
"messages": [{"role":"user","content":
"Write a Python function binary_search(arr, target) returning the index or -1. Code only, in a ```python block."}]}
```

**TP=2 output** (note the bare `else` with no colon — the file will not parse):

```python
elif arr[mid] < target:
low = mid + 1
else
high = mid - 1
```

**TP=1 output**, same prompt, same seed, same everything else:

```python
else:
high = mid - 1
```

`if`, `elif`, `while` and `def` all keep their colons in both cases. It is
specifically wrong tokens, not malformed formatting.

## The token distribution is corrupted, not merely unlucky

With `logprobs`, at the position immediately after the token `else`:

| Config | chosen | logprob of chosen | logprob of `:` |
|---|---|---|---|
| TP=1 | `:` | −0.0009 | **−0.0009** |
| TP=2 | `\n` | −0.0000 | **−13.9961** |

The model is ~100% confident in the wrong token, in a context where the same
weights on the same machine are 99.9% confident in the correct one. For
reference, the same server on `/v1/completions` prompted with a raw
`...else` prefix picks `:` at −0.0551.

## Evidence matrix

Instrument: 8 Python codegen prompts, temperature 0, extracted from
```python fences and checked with `ast.parse()`.

| # | Config | MoE local width | 128-aligned | Kernel | Result |
|---|---|---|---|---|---|
| 1 | TP=1 | 704 | no | TRITON | **8/8 clean** |
| 2 | TP=2 | 352 | no | TRITON | 6/8 |
| 3 | TP=2, `max_model_len` 4096 / util 0.95 | 352 | no | TRITON | 6/8, byte-identical token counts to #2 |
| 4 | TP=2 + `CCL_ENABLE_SYCL_KERNELS=0` | 352 | no | TRITON | 6/8, byte-identical to #2 |
| 5 | TP=2 + `--enable-expert-parallel` | — | — | TRITON | deadlock: `TimeoutError: RPC call to sample_tokens timed out` → `EngineDeadError` |
| 6 | **PP=2** | 704 | no | TRITON | **8/8 clean**, token counts bit-identical to #1 |
| 7 | Qwen3-8B **dense** fp16, TP=2 | — | — | — | **8/8 clean** |
| 8 | Qwen3-Coder-30B-A3B **MoE** fp8, TP=2 | 384 | yes | **XPU** | **8/8 clean** |
| 9 | **padded 704→768**, PP=2 (control) | 768 | yes | TRITON | **8/8 clean** |
| 10 | **padded 704→768, TP=2** | **384** | **yes** | TRITON | **7/8 — same defect** |

## What this is not

**It is not the 128-alignment problem.** The usual account is that
`704 / 2 = 352`, `352 % 128 = 96`, so the fused MoE GEMM is unhappy, and the
remedy is an offline zero-pad of `moe_intermediate_size` to 768 (as shipped by
several community quantizers). I built that padded checkpoint — gate and up
halves padded independently within the fused `gate_up_proj`, `down_proj`
padded on its trailing axis, `config.json` updated — validated it under PP=2
(row 9: 8/8 clean), and then ran it at TP=2.

**Row 10: the identical defect reproduces at a perfectly aligned 384-wide
local shard.** Padding moved the `:` logprob from −13.9961 to −10.6563 and
recovered one task, but the model is still confidently wrong.

I think the alignment story is real *on CUDA*, where Marlin/CUTLASS raise a
hard assert and padding converts a crash into a working config. On XPU there
is no assert: Triton accepts the ragged K via `tl.cdiv` tail-masking and
computes wrong values instead. Padding fixes the CUDA assert; it does not fix
the XPU numerics.

**It is not the collective transport.** `CCL_ENABLE_SYCL_KERNELS=0` — the only
stable profile reported in #41663 — produced *byte-identical* corrupt output
(row 4). If the all-reduce were corrupting, changing the collective kernel
implementation should perturb the result. It did not move a single token.

**It is not capacity or scheduling.** Rows 2 and 3 differ in
`max_model_len` and `gpu_memory_utilization` and are byte-identical.

**It is not the model or the quantization.** Rows 1 and 6 are clean on the
same weights at the same precision.

## Suspected root cause

The Triton FP8 fused-MoE kernel on the XPU platform is numerically incorrect
when the expert weights are tensor-sharded, independent of shard alignment.
TP=1 and PP=2 — neither of which tensor-shards the experts — are both clean.

A secondary observation that may help triage: **the native `XPU` FP8 MoE
backend declines Gemma 4 at both 704 and 768, and at both TP=1 and TP=2**, so
its selection predicate is not alignment. Gemma 4 therefore always falls
through to Triton on this stack, while Qwen3-Coder-30B-A3B (row 8) is
accepted by the `XPU` backend and never exercises the broken path. That is
probably why the alignment correlation has looked causal.

## Open caveat

I padded only the MoE. The dense/shared MLP (`intermediate_size=2112` →
1056 per rank) is also not 128-aligned and is still tensor-sharded in row 10,
so I cannot yet exclude it as a contributor. Row 7 (dense fp16 at TP=2, clean)
is weak counter-evidence since it is a different dtype path. Happy to pad the
dense MLP to 2304 and re-run if that would help.

## Note on detecting this class of defect

The thing that makes this one awkward is that it is invisible to the checks
you would normally reach for. The server is healthy, the finish reason is
`stop`, and the prose is fluent — a smoke test that asks "did it answer
coherently?" passes on the corrupting endpoint. Our own prior multi-card
qualification for this hardware ("5/5 smoke prompts correct, all
`finish_reason=stop`") passes on it too; that is how we shipped a TP=2
config we believed was fine.

What actually surfaced it was a machine-checkable predicate — generate code,
run `ast.parse()` on it — plus a logprob spot-check at a position where the
correct token is known. Both are cheap. Offering it in case it is useful for
the model validation matrix, since any TP correctness check based on output
plausibility alone will not catch this.

## Related issues in this repo

I looked for a duplicate before filing. The closest are related but distinct:

- **#603** (Qwen3.6-27B dense, sym_int4, TP=2, 2× B50) — runaway `!`
degeneration. Same family (TP=2 output corruption) but *visible* garbage,
a different model and quant, and the polarity is inverted: #603 reports
`0.21.0-b1` as the **good** image, whereas b1 is the image corrupting here.
So this is not the same regression.
- **#534** (Gemma 4 12B BF16, Arc Pro B70) — corrupted output on the **same
image digest** `sha256:5d87be271e4d…`. Closed as completed. Different
symptom (gibberish within 1–2 tokens) and a dense model, but it may share a
root cause worth checking against this report.
- **#544** (gemma-4-31B, 4× B70, TP=4, fp8 and sym_int4) — "timeout or
repeated answers", closed with the guidance that sym_int4 on 2 × B70 works.
Different model (31B dense) and a different symptom, but it is the nearest
neighbour on multi-GPU Gemma 4 + B70, so I mention it in case the two turn
out to be connected.
- **#532** (k_descale stride assertion on Gemma 4) — different, an assert.
- **#424** (`UR_RESULT_ERROR_DEVICE_LOST` at TP=2) — I also hit
`UR_RESULT_ERROR_DEVICE_LOST` once, but only under extreme VRAM pressure
(27.6 GiB of weights at `--gpu-memory-utilization 0.95` on a 32 GB card).
That looks like a separate memory-pressure issue, not this one.

## Workarounds that do work

- `--pipeline-parallel-size 2` instead of TP — clean, and full 262,144
context with 801,746 KV tokens on 2 × 32 GB.
- Single card with Google's QAT checkpoint
(`gemma-4-26B-A4B-it-qat-q4_0-unquantized`) + `--quantization sym_int4` —
clean, 16.44 GiB weights, full 262,144 context, and faster than the
two-card FP8 configuration.