microsoft / microsoft/onnxruntime
onnruntime_shared_lib_test.exe passes/fails in non-deterministic fashion
@tianleiwu is already working on this.
Since Jun 16, 2026.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
As the title says. I have a fresh build of 1.26 in RelWithDebInfo, with the CUDA EP, and onnruntime_shared_lib_test.exe fails about 40% of the time.
Looking at the definition of MyCustomKernelSecondInputOnCpu::Compute in custom_op_utils.cc, it seems to me that cudaMemcpyAsync shoud be used in line 101, instead of cudaMemcpy, to properly sequence the memcpy operation with the add operation. Otherwise, if the stream is non-blocking (as is the case in this test), the add kernel might run before memcpy has completed. Indeed, after changing the code that way and rebuilding, I don't observe failures anymore.
### To reproduce
Build 1.26 in RelWithDebInfo with the CUDA EP, then run `onnxruntime_shared_lib_test.exe --gtest_filter=CApiTest.custom_op_set_input_memory_type --gtest_repeat=20`.
### Urgency
_No response_
### Platform
Windows
### OS Version
11
### ONNX Runtime Installation
Built from Source
### ONNX Runtime Version or Commit ID
1.26
### ONNX Runtime API
C++
### Architecture
X64
### Execution Provider
CUDA
### Execution Provider Library Version
CUDA 12.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.