microsoft / microsoft/onnxruntime
CUDA EP: Slice with a runtime bound returns its input untrimmed, breaking a downstream ScatterND
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
On the CUDA execution provider, a `Slice` whose `ends` is computed at runtime from another input's shape returns its input **unmodified**. The graph runs correctly on the CPU EP.
The construct is what `torch.onnx` emits for a prefix slice-assignment `buf[:S] = updates`:
```
indices = Unsqueeze( Slice( Range(0, T, 1), starts=[0], ends=[S] ), -1 )
out = ScatterND(data, indices, updates)
```
`S` comes from `Shape(y)[1]`, so `indices` is `S` long by construction and agrees with `updates`. On CUDA the `Slice` trims nothing, `indices` arrives with `T` entries, and `ScatterND` rejects the shapes:
```
Non-zero status code returned while running ScatterND node. Name:'node_ScatterND_101'
Status Message: updates tensor should have shape equal to
indices.shape[:-1] + data.shape[indices.shape[-1]:].
updates shape: {60,7,1,128}, indices shape: {70,1}, data shape: {70,7,1,128}
```
60 is `S`, 70 is `T`. The returned tensor is the whole `Range` (`[0,1,…,T-1]`), not garbage — it is read back intact, so this looks like the bound being read from a buffer whose contents are no longer valid rather than a corrupted result.
### Isolation
Exposing one intermediate at a time as a **graph output** (graph outputs are excluded from buffer reuse) on a reduced 68-node cut:
| exposed as an output | CUDA |
|---|---|
| *(none)* | FAIL |
| the `T` scalar | FAIL |
| **the `S` scalar** | **OK** |
| **`ends`** | **OK** |
| the `Range` / `Slice` / `indices` / `data` tensors | FAIL |
Only the tensors carrying `S` change the outcome. `enable_cpu_mem_arena=False` also fixes the reduced cut, though not the full graph. That points at the CPU-side buffer holding `S` being recycled before the CUDA `Slice` kernel reads it.
A minimal hand-built graph of the same shape (`Shape → Squeeze → Unsqueeze → Slice`) does **not** reproduce, so it appears to need the surrounding allocation pattern.
### To reproduce
Self-contained; the graph is a public weight-free ONNX file, and the script synthesizes initializers for it.
```bash
pip install onnxruntime-gpu==1.23.2 onnx numpy 'nvidia-cudnn-cu12<10'
curl -O https://raw.githubusercontent.com/DataZooDE/anofox-tabfm/main/tools/gpu_test/ort_repro.py
python ort_repro.py # pre-fix graph: CPU ok, CUDA fails
python ort_repro.py --pinned # bounds named as graph outputs: both ok
```
Observed:
```
onnxruntime 1.23.2 graph v2026.08.13
T=70 S=60
CPUExecutionProvider OK logits (1, 70, 10)
CUDAExecutionProvider FAIL ... ScatterND ... updates {60,7,1,128}, indices {70,1}
T=128 S=100
CPUExecutionProvider OK logits (1, 128, 10)
CUDAExecutionProvider FAIL ... updates {100,12,1,128}, indices {128,1}
onnxruntime 1.23.2 graph v2026.08.14 (bounds pinned as graph outputs)
CPUExecutionProvider OK
CUDAExecutionProvider OK
```
### Urgency
Not urgent for us — naming the two bound tensors as graph outputs works around it, and that is what we ship. Filing because the workaround is load-bearing for a reason nobody would guess from reading the graph, and because anything `torch.onnx` exports from a prefix slice-assignment can hit this.
### Platform
Linux
### OS Version
Ubuntu 22.04 (container)
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.22.0, 1.23.2, 1.24.2, 1.25.1, 1.26.0, 1.28.0 — all reproduce identically
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
CUDA
### Execution Provider Library Version
CUDA 12.4 / cuDNN 9; reproduced on RTX 3060, RTX 3070 and RTX A5000 (all sm_86), driver 570/580
Contributor guide
Research direction
Start with the provided tools/gpu_test/ort_repro.py reproduction and compare the CUDA results with and without the bound tensors exposed as graph outputs. Investigate the CUDA Slice path and the surrounding buffer-reuse behavior for runtime ends values. Done means the unmodified Slice result and downstream ScatterND shapes match the CPU path without requiring graph-output pinning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100