[ROCm] Out-of-bounds gather/index_select never completes on gfx1103: no assert, next sync hangs unkillably
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🐛 Describe the bug
On gfx1103 (Radeon 780M), passing out-of-bounds indices to `Tensor.gather` or `torch.index_select` leaves the HIP stream permanently un-drained. Nothing is raised — no device-side assert, empty stderr, the op returns normally because it is asynchronous.
The next blocking synchronization then spins forever inside `c10::cuda::memcpy_and_sync` → `hipMemcpyWithStream` → `SwitchToThread`. At that point the process cannot be killed (`Stop-Process -Force` and `taskkill /F /T` both report success while the process stays alive) and Windows cannot shut down — `shutdown /r /f` is accepted but never completes, and `shutdown /a` returns "A system shutdown is in progress. (1115)". Only a hard power-off recovers the machine.
The same indices on CPU raise immediately:
```
RuntimeError: index 156073 is out of bounds for dimension 0 with size 51865
```
### Repro (no model, no data)
```python
import time
import torch
V, T = 51865, 444
g = torch.Generator().manual_seed(0)
tokens = torch.randint(0, V, (1, T), generator=g)
beams = torch.randint(0, 5, (1, T), generator=g) # zeros_like(beams) for the control
scores = torch.randn(V, T, device="cuda", dtype=torch.bfloat16)
idx = (tokens + beams * V).cuda() # 339/444 out of bounds, max 258,673, domain 51,865
out = scores.gather(0, idx) # returns immediately: async, proves nothing
st = torch.cuda.current_stream()
t0 = time.time()
while time.time() - t0 < 60: # non-blocking; never becomes True
if st.query():
print("drained after", round(time.time() - t0, 2), "s")
break
time.sleep(0.05)
else:
print("stream still not drained after 60 s")
# out.cpu() # <-- what a real program does next. Hangs unkillably; needs a hard power-off.
```
With `beams` replaced by `torch.zeros_like(beams)` the indices are all in range and `st.query()` is True after 0.05 s.
⚠️ The last line is commented out on purpose. We observed this through the non-blocking `query()` precisely so we would not have to power-cycle the machine again; every run below exits cleanly and leaves nothing behind.
### Measured
| op | indices | `stream.query()` |
|---|---|---|
| `Tensor.gather` | in bounds | True after **0.05 s** (2 polls) |
| `Tensor.gather` | **339/444 out of bounds** | **False for 420.03 s** (8,254 polls) |
| `torch.index_select` | in bounds | True after 0.05 s |
| `torch.index_select` | **out of bounds** | **False for 30.02 s** (578 polls) |
| `F.embedding` | in bounds | True after 0.05 s |
| `F.embedding` | **out of bounds** | True after **20.66 s** |
`F.embedding` is the interesting row: with the same out-of-bounds indices it does complete, roughly 400x slower than the in-bounds control. So this is not simply "the faulting kernel kills the queue" for every op — the behaviour differs per operator, and for `gather` we could not observe completion at all.
### How we hit it
`transformers` computes out-of-bounds gather indices for Whisper beam search (huggingface/transformers#48621 — confirmed by a maintainer, fix pending). On CPU that surfaces immediately as the `RuntimeError` above. On this GPU it silently wedges the whole machine, which took us several days to attribute because the failure looks nothing like a bad index.
We are not asking for the indices to be validated on the fast path. The gap that cost us is that an out-of-bounds access here produces **no diagnostic of any kind** — a device-side assert, or a HIP error surfaced at the next sync, would have made this a five-minute bug.
### Known limits of these measurements
- One machine, one GPU, one build. We have no NVIDIA GPU to compare against.
- "Never drains" means "not observed to drain within 420 s", not a proof that it never would.
- The blocking variant (`out.cpu()`) has been observed in a real workload — that is where the py-spy stack and the failed shutdown come from — but we did not re-run it deliberately.
### Note on `collect_env`
`python -m torch.utils.collect_env` crashes on this machine before printing anything:
```
File "torch/utils/collect_env.py", line 564, in get_windows_version
...
File "encodings/oem.py", line 15, in decode
UnicodeDecodeError: 'cp1' codec can't decode bytes in position 0--1: No mapping for
the Unicode character exists in the target code page.
```
It decodes `powershell.exe` output with the OEM codepage, which fails on a zh-TW Windows. Fields collected by hand instead.
### Versions
```
torch : 2.12.0+rocm7.14.0
is_debug : False
hip : 7.14.60850
python : 3.13.12
platform : Windows-11-10.0.26200-SP0
gpu : AMD Radeon Graphics, gfx1103, 6.2 GiB
driver : 32.0.31007.6002
transformers : 5.14.1
```
cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @jataylo @hongxiayang @naromero77amd @pragupta @jerrymannil @xinyazhang
Contributor guide
Assessment
This issue has not been assessed yet.