intel / intel/torch-xpu-ops

[BUG][PVC/BMG] gather kernel out-of-bound assert in OmniVoice TTS model.generate() — CPU PASS, XPU FAIL

Open
#5,115 5 comments 0 reactions 2 assignees Claimed by @lchen2331 View on GitHub
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

## Environment
- GPU: Intel Data Center GPU Max 1100 (PVC)
- torch: 2.15.0.dev20260824+xpu (latest nightly, also reproduced on dev20260812)
- triton-xpu: 3.8.0+git1e2d42a0
- Python: 3.10.20
- OS: Linux x86_64
- Model package: `pip install omnivoice` (k2-fsa/OmniVoice TTS)

## Minimal Reproducer

```python
import torch, numpy as np
from omnivoice import OmniVoice

REF_WAV = "harvard.wav" # any ~18s speech WAV
REF_TEXT = "The birch canoe slid on the smooth planks."
TEXT = "Hello."

# CPU: PASS
model_cpu = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="cpu", dtype=torch.float32)
audio_cpu = model_cpu.generate(text=TEXT, ref_audio=REF_WAV, ref_text=REF_TEXT)
assert len(audio_cpu[0]) > 0
print(f"CPU: PASS ({len(audio_cpu[0])} samples)")
del model_cpu

# XPU: FAIL — device-side assert terminates process
model_xpu = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="xpu:0", dtype=torch.float32)
audio_xpu = model_xpu.generate(text=TEXT, ref_audio=REF_WAV, ref_text=REF_TEXT)
print(f"XPU: PASS ({len(audio_xpu[0])} samples)")
```

Reference audio can be any speech WAV file (~6-18 seconds).

## Expected Behavior

Generation completes and produces valid audio samples (as it does on CPU).

## Actual Behavior

Device-side assertion terminates process:

```
/__w/pytorch/pytorch/third_party/torch-xpu-ops/src/ATen/native/xpu/sycl/IndexKernelUtils.h:62:
operator(): global id: [...], local id: [...]
Assertion `ind >= 0 && ind < ind_dim_size_ && "vectorized gather kernel index out of bounds"` failed.
```

## What We Ruled Out

| Test | Result |
|------|--------|
| CPU generation | ✅ Always passes |
| topk indices in bounds | ✅ Verified (all idx < input_numel) |
| Tensor dimensions match | ✅ Verified |
| Isolated topk + advanced indexing put (5000 iterations) | ✅ Does NOT reproduce |
| Adding `torch.xpu.synchronize()` between ops | ❌ Does NOT prevent crash |
| Replacing advanced indexing put with `index_select()` + `scatter()` at one call site | ❌ Does NOT fully fix |
| Different transformers version (5.12 vs 5.16) | ❌ Still crashes |
| Different GPU card | ❌ Still crashes |
| FP16 / BF16 / FP32 | ❌ All crash |

## Analysis

The OOB gather originates from a SYCL kernel dispatch inside the model forward pass. The exact source is difficult to isolate because:

1. The crash is intermittent (sometimes succeeds, sometimes asserts)
2. Any Python-level instrumentation (hooks, `.cpu()`, thread spawn) prevents reproduction by changing kernel scheduling
3. The crash occurs inside `model.generate()` which dispatches hundreds of kernels

We suspect a missing dependency barrier between dependent kernel launches in the SYCL command queue, causing one kernel to read uninitialized data from another.

## Request

Help identifying which specific SYCL kernel dispatch triggers the OOB gather. Debug tracing of `IndexKernelUtils.h:62` call sites would help narrow down the root cause.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.