microsoft / microsoft/onnxruntime

onnruntime_shared_lib_test.exe passes/fails in non-deterministic fashion

Open
#28,956 0 comments 1 reaction 2 assignees View on GitHub

@tianleiwu is already working on this.

Since Jun 16, 2026.

Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

As the title says. I have a fresh build of 1.26 in RelWithDebInfo, with the CUDA EP, and onnruntime_shared_lib_test.exe fails about 40% of the time.

Looking at the definition of MyCustomKernelSecondInputOnCpu::Compute in custom_op_utils.cc, it seems to me that cudaMemcpyAsync shoud be used in line 101, instead of cudaMemcpy, to properly sequence the memcpy operation with the add operation. Otherwise, if the stream is non-blocking (as is the case in this test), the add kernel might run before memcpy has completed. Indeed, after changing the code that way and rebuilding, I don't observe failures anymore.

### To reproduce

Build 1.26 in RelWithDebInfo with the CUDA EP, then run `onnxruntime_shared_lib_test.exe --gtest_filter=CApiTest.custom_op_set_input_memory_type --gtest_repeat=20`.

### Urgency

_No response_

### Platform

Windows

### OS Version

11

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

1.26

### ONNX Runtime API

C++

### Architecture

X64

### Execution Provider

CUDA

### Execution Provider Library Version

CUDA 12.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.