CUDA failure 101 (invalid device ordinal, GPU=-1) in NonZero CUDA kernel on Linux

Describe the issue

The CUDA Execution Provider's NonZero kernel fails with CUDA failure 101: invalid device ordinal ; GPU=-1 on every inference call on Linux. The error occurs at nonzero_op.cc:71 during NonZeroCalcPrefixSumTempStorageBytes. All other CUDA operations (Conv, MatMul, etc.) execute successfully in the same session on the same GPU.

The error is non-fatal — ONNX Runtime appears to handle it internally and inference completes, but the NonZero node produces incorrect/empty results.

This issue does not reproduce with the Windows Microsoft.ML.OnnxRuntime.Gpu package using the same model, same ONNX Runtime version (1.24.4), and same GPU hardware (tested on NVIDIA RTX A2000).

Error message

[E:onnxruntime:CSharpOnnxRuntime, cuda_call.cc:123 CudaCall] CUDA failure 101: invalid device ordinal ; GPU=-1 ; hostname=5d632b12d8de45ae90de8a55274557a4 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/tensor/nonzero_op.cc ; line=71 ; expr=NonZeroCalcPrefixSumTempStorageBytes(Stream(context), prefix_counts, number_of_blocks, temp_storage_bytes);

[E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running NonZero node. Name:'NonZero_2221' Status Message: CUDA failure 101: invalid device ordinal ; GPU=-1

To reproduce

Summary

• 100% reproducible on every inference call — not intermittent
• Occurs during the NonZero_2221 node execution (post-processing NMS in a YOLOv7 pose estimation model)
• Fails on first inference and every subsequent inference
• All other CUDA operations in the same session work correctly

Important: NOT reproducible with standalone NonZero model

A minimal ONNX model containing only a NonZero op runs correctly on CUDA EP with all tensor sizes tested (up to 25200x57 — the typical YOLOv7 NMS output). The error only manifests when NonZero executes within the context of the full YOLOv7 model graph, suggesting the issue is related to CUDA stream/context state from preceding kernel executions (Conv, MatMul, etc.) rather than the NonZero kernel itself.

We can provide the model privately if needed for reproduction.

Model details

• YOLOv7 pose estimation model (ONNX format, ~308 MB)
• Model contains NonZero operations used in NMS (Non-Maximum Suppression) post-processing
• Input shape: (1, 3, 960, 960)
• The NonZero_2221 node is deep in the post-processing subgraph, after hundreds of Conv/MatMul/Sigmoid ops have executed on CUDA EP

Session configuration

var sessionOptions = new SessionOptions();
using var cudaProviderOptions = new OrtCUDAProviderOptions();
cudaProviderOptions.UpdateOptions(new Dictionary<string, string>
{
    ["device_id"] = "0",
    ["gpu_mem_limit"] = "8589934592",
    ["arena_extend_strategy"] = "kNextPowerOfTwo",
    ["cudnn_conv_algo_search"] = "EXHAUSTIVE",
    ["do_copy_in_default_stream"] = "1",
    ["cudnn_conv_use_max_workspace"] = "1",
    ["cudnn_conv1d_pad_to_nc1d"] = "1",
});
sessionOptions.AppendExecutionProvider_CUDA(cudaProviderOptions);
sessionOptions.IntraOpNumThreads = 1;
var session = new InferenceSession(modelBytes, sessionOptions);

Verbose ORT logging confirms

• CUDA EP claims the NonZero node during session initialization (it is NOT assigned to CPU fallback)
• ORT's fallback_cpu_capability.cc intentionally moves shape-related ops (Gather, Slice, Concat, Where, Equal, Expand) to CPU, but NonZero is NOT among them
• The NonZero node stays on CUDA EP and fails at runtime

Environment

Fails (Linux container)

Component	Version
OS	Ubuntu 24.04.1 LTS (Noble) in Docker
ONNX Runtime	1.24.4 (Microsoft.ML.OnnxRuntime.Gpu.Linux NuGet)
.NET	10.0.4
CUDA Runtime	12.6.77 (libcudart.so.12.6.77)
cuDNN	9.5.1 (libcudnn.so.9.5.1)
cuBLAS	12.6.4.1
TensorRT	10.7.0.23 (cuda12.6) — not used, EP disabled
Host Driver	573.44 (CUDA 12.8 capable)
Docker base image	nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04

Tested on GPUs:

• NVIDIA RTX A2000 (Compute 8.6, 6GB) — Docker Desktop on Windows host
• NVIDIA Tesla T4 (Compute 7.5, 16GB) — Azure Container Apps

Works (Windows)

Component	Version
OS	Windows 11 Pro
ONNX Runtime	1.24.4 (Microsoft.ML.OnnxRuntime.Gpu.Windows NuGet)
GPU	NVIDIA RTX A2000 (same hardware as Linux test)

Same model, same session configuration, same GPU — no NonZero error on Windows.

Dependency verification

All CUDA libraries resolve correctly in the Linux container (ldd shows no missing dependencies):

libcudart.so.12  -> /usr/local/cuda-12.6/targets/x86_64-linux/lib/libcudart.so.12.6.77
libcublas.so.12  -> /usr/local/cuda-12.6/targets/x86_64-linux/lib/libcublas.so.12.6.4.1
libcublasLt.so.12 -> /usr/local/cuda-12.6/targets/x86_64-linux/lib/libcublasLt.so.12.6.4.1
libcudnn.so.9    -> /usr/lib/x86_64-linux-gnu/libcudnn.so.9.5.1
libcufft.so.11   -> /usr/local/cuda-12.6/targets/x86_64-linux/lib/libcufft.so.11.3.0.4
libcurand.so.10  -> /usr/local/cuda-12.6/targets/x86_64-linux/lib/libcurand.so.10.3.7.77

Workaround

Enabling the TensorRT Execution Provider before the CUDA EP causes TensorRT to fuse the NonZero operation into optimized NMS kernels, avoiding the buggy CUDA EP NonZero implementation entirely.

Attempted fixes (none resolved the issue)

Change	Result
do_copy_in_default_stream = "0" (use ORT's own CUDA stream)	Still fails
GraphOptimizationLevel.ORT_ENABLE_BASIC (disable extended optimizations)	Still fails
arena_extend_strategy = "kNextPowerOfTwo" + cudnn_conv_algo_search = "EXHAUSTIVE"	Fixed Conv fallback warnings but NonZero still fails
Standalone NonZero model (all tensor sizes up to 25200x57)	Works correctly — bug is graph-context-dependent

Analysis

The GPU=-1 in the error message suggests the CUDA EP's internal device tracking returns an invalid device ordinal when executing the NonZero kernel's cub::DeviceScan (via NonZeroCalcPrefixSumTempStorageBytes). Since all other CUDA operations succeed on the same device in the same session, this appears to be specific to the NonZero kernel's CUDA stream/context handling on Linux.

A standalone NonZero model runs correctly on CUDA EP at all tensor sizes (tested up to 25200x57), so the issue is not in the NonZero kernel in isolation. It only manifests when NonZero executes within a large YOLOv7 model graph after hundreds of prior CUDA kernel executions (Conv, MatMul, Sigmoid, etc.). This suggests the NonZero kernel inherits corrupted stream/device state from preceding kernel executions within the same ORT session.

The issue is not caused by external CUDA context interference (e.g., OpenCV DNN CUDA), session configuration options, or graph optimization level — these were all tested and ruled out.

The fact that the Windows build works with identical configuration on the same GPU hardware suggests a Linux-specific build issue in the CUDA EP's NonZero kernel or its interaction with the execution framework on Linux.

Urgency

No response

Platform

Linux

OS Version

Ubuntu 24.04.1 LTS (Noble) in Docker

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.24.4

ONNX Runtime API

C#

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA Runtime 12.6.77