microsoft / microsoft/onnxruntime

onnxruntime with CUDAExecutionProvider crashes: gather_nd.cc:30 CheckBatchDimensionsMatch Batch dimensions differ at index 0: 1 != 3, tensor indices: 0, 1

Open
#25,053 0 comments 1 reaction 1 assignee Assigned to @Copilot View on GitHub
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

For the following onnx model,

![Image](https://github.com/user-attachments/assets/658f575b-b204-4c7d-89d2-37a5ab009007)
onnxruntime with the CPUExecutionProvider can run this model correctly, the results are as follows:
```c
ONNXRuntime:
[array([[0.62514794],
[0.06079907],
[1. ]], dtype=float32)]
```
However, when I run this model with CUDAExecutionProvider, onnxruntime crashes:
```c
File "/home/carla/anaconda3/envs/onnruntime-gpu/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 273, in run
return self._sess.run(output_names, input_feed, run_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running GatherND node. Name:'GatherND' Status Message: gather_nd.cc:30 CheckBatchDimensionsMatch Batch dimensions differ at index 0: 1 != 3, tensor indices: 0, 1
```

### To reproduce

### Environment
OS: Ubuntu 20.04
onnxruntime: 1.23.0.dev20250515001
CUDA: cuda-12.2.2::cuda-toolkit
CUDNN: 9.1.1.17
NVIDIA GPU: GeForce RTX 3080
NVIDIA Driver Version: 535.183.01
Python Version: 3.12.9

### Steps to reproduce
This bug can be reproduced by the following code with the model in the attachment.
```python
from typing import Dict, List, Literal, Optional
import sys
import os

import numpy as np
import onnx
import onnxruntime
from onnx import ModelProto, TensorProto, helper, mapping

import pickle

def test():

onnx_model = onnx.load("222.onnx")
print(onnx_model.opset_import[0].version)

with open("inputs.pkl", "rb") as fp:
inputs = pickle.load(fp)

ort_session = onnxruntime.InferenceSession(
onnx_model.SerializeToString(), providers=["CPUExecutionProvider"]
)
ort_output = ort_session.run([], inputs)

print("ONNXRuntime:\n", ort_output)

#--------------------------------------------

ort_session = onnxruntime.InferenceSession(
onnx_model.SerializeToString(), providers=["CUDAExecutionProvider"]
)
ort_output = ort_session.run([], inputs)

print("ONNXRuntime:\n", ort_output)

if __name__ == "__main__":
test()
```

[testcase.zip](https://github.com/user-attachments/files/20727403/testcase.zip)

### Urgency

_No response_

### Platform

Linux

### OS Version

Ubuntu 20.04

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.23.0.dev20250515001

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

Default CPU, CUDA

### Execution Provider Library Version

CUDA: cuda-12.2.2::cuda-toolkit CUDNN: 9.1.1.17

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.