microsoft / microsoft/onnxruntime

[Bug] Wrong Conv behavior using CUDAExecutionProvider in C# when kernel_shape=2

Open
#26,215 2 comments 1 reaction 1 assignee Assigned to @Copilot View on GitHub
api:CSharp ep:CUDA
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

I previously worked with Microsoft.ML.Onnxruntime.GPU 1.16.0 CUDAExecutionProvider, and everything ran fine when using a Conv layer with kernel_shape=2.
However, after upgrading to 1.22.0, the output of my segmentation model became incorrect. The model still executes without errors, but the output image shows a completely wrong pattern.

If I change the Conv layer to use kernel_shape=3 and retrain the model, the results turn out correct again. Also, If I keep kernel_shape=2 but switch to CPUExecutionProvider, the results turn out correct. This makes me suspect there’s an issue with kernel_shape=2 in the newer version of CUDAExecutionProvider.

In my use case, I need to maintain backward compatibility for existing models, so simply switching from kernel_shape=2 to 3 is not an option.

### To reproduce

build a segmentation model which contains Conv:
dialations: 1,1
kernel_shape: 2,2
pads: 0,0,1,1
strides: 1,1

### Urgency

_No response_

### Platform

Windows

### OS Version

22H2

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.22.0

### ONNX Runtime API

C#

### Architecture

X64

### Execution Provider

CUDA

### Execution Provider Library Version

12.8

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.