[BUG][PVC/BMG] gather kernel out-of-bound assert in OmniVoice TTS model.generate() — CPU PASS, XPU FAIL
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
## Environment
- GPU: Intel Data Center GPU Max 1100 (PVC)
- torch: 2.15.0.dev20260824+xpu (latest nightly, also reproduced on dev20260812)
- triton-xpu: 3.8.0+git1e2d42a0
- Python: 3.10.20
- OS: Linux x86_64
- Model package: `pip install omnivoice` (k2-fsa/OmniVoice TTS)
## Minimal Reproducer
```python
import torch, numpy as np
from omnivoice import OmniVoice
REF_WAV = "harvard.wav" # any ~18s speech WAV
REF_TEXT = "The birch canoe slid on the smooth planks."
TEXT = "Hello."
# CPU: PASS
model_cpu = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="cpu", dtype=torch.float32)
audio_cpu = model_cpu.generate(text=TEXT, ref_audio=REF_WAV, ref_text=REF_TEXT)
assert len(audio_cpu[0]) > 0
print(f"CPU: PASS ({len(audio_cpu[0])} samples)")
del model_cpu
# XPU: FAIL — device-side assert terminates process
model_xpu = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="xpu:0", dtype=torch.float32)
audio_xpu = model_xpu.generate(text=TEXT, ref_audio=REF_WAV, ref_text=REF_TEXT)
print(f"XPU: PASS ({len(audio_xpu[0])} samples)")
```
Reference audio can be any speech WAV file (~6-18 seconds).
## Expected Behavior
Generation completes and produces valid audio samples (as it does on CPU).
## Actual Behavior
Device-side assertion terminates process:
```
/__w/pytorch/pytorch/third_party/torch-xpu-ops/src/ATen/native/xpu/sycl/IndexKernelUtils.h:62:
operator(): global id: [...], local id: [...]
Assertion `ind >= 0 && ind < ind_dim_size_ && "vectorized gather kernel index out of bounds"` failed.
```
## What We Ruled Out
| Test | Result |
|------|--------|
| CPU generation | ✅ Always passes |
| topk indices in bounds | ✅ Verified (all idx < input_numel) |
| Tensor dimensions match | ✅ Verified |
| Isolated topk + advanced indexing put (5000 iterations) | ✅ Does NOT reproduce |
| Adding `torch.xpu.synchronize()` between ops | ❌ Does NOT prevent crash |
| Replacing advanced indexing put with `index_select()` + `scatter()` at one call site | ❌ Does NOT fully fix |
| Different transformers version (5.12 vs 5.16) | ❌ Still crashes |
| Different GPU card | ❌ Still crashes |
| FP16 / BF16 / FP32 | ❌ All crash |
## Analysis
The OOB gather originates from a SYCL kernel dispatch inside the model forward pass. The exact source is difficult to isolate because:
1. The crash is intermittent (sometimes succeeds, sometimes asserts)
2. Any Python-level instrumentation (hooks, `.cpu()`, thread spawn) prevents reproduction by changing kernel scheduling
3. The crash occurs inside `model.generate()` which dispatches hundreds of kernels
We suspect a missing dependency barrier between dependent kernel launches in the SYCL command queue, causing one kernel to read uninitialized data from another.
## Request
Help identifying which specific SYCL kernel dispatch triggers the OOB gather. Debug tracing of `IndexKernelUtils.h:62` call sites would help narrow down the root cause.
Contributor guide
Assessment
This issue has not been assessed yet.