microsoft / microsoft/onnxruntime

CUDA EP ReduceMax/ReduceMin return finite limits instead of +/-inf for all-infinite inputs

Open
#32,256 1 comment 0 reactions 0 assignees View on GitHub
ep:CUDA
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

### Describe the issue

For floating-point reductions, CUDAExecutionProvider returns finite float limits when all reduced values are infinite, while CPUExecutionProvider and TensorRTExecutionProvider return IEEE +/-inf.

Minimal cases:

- `ReduceMax([-inf, -inf])`: CPU EP returns `-inf`, TensorRT EP returns `-inf`, but CUDA EP returns `-3.4028235e+38`.
- `ReduceMin([+inf, +inf])`: CPU EP returns `+inf`, TensorRT EP returns `+inf`, but CUDA EP returns `+3.4028235e+38`.

This looks like a numerical correctness issue in the CUDA reduction path for valid IEEE floating-point inputs. The result is stable across repeated runs.

I reproduced the issue with:

- onnxruntime-gpu 1.17.1
- onnxruntime-gpu 1.18.1
- Python API
- Linux x86_64
- CUDA 11.8 / cuDNN 8.6
- NVIDIA GeForce RTX 3080 Ti

I also tried checking onnxruntime-gpu 1.23.2, but that wheel requires CUDA 12 / cuDNN 9 and falls back to CPU on this machine, so I could not validate the latest CUDA EP behavior here.

### To reproduce

### To reproduce

Run the following script on a machine with CUDAExecutionProvider available:

```python
import numpy as np
import onnx
import onnxruntime as ort
from onnx import TensorProto, helper

print("onnxruntime", ort.__version__)
print("available providers", ort.get_available_providers())

def make_model(op):
model = helper.make_model(
helper.make_graph(
[helper.make_node(op, ["x"], ["z"], axes=[0], keepdims=0)],
op,
[helper.make_tensor_value_info("x", TensorProto.FLOAT, [2])],
[helper.make_tensor_value_info("z", TensorProto.FLOAT, [])],
),
opset_imports=[helper.make_operatorsetid("", 17)],
)
model.ir_version = 8
onnx.checker.check_model(model)
return model

def run(model, feed, providers, opt):
so = ort.SessionOptions()
so.graph_optimization_level = opt
sess = ort.InferenceSession(model.SerializeToString(), so, providers=providers)
return sess.get_providers(), sess.run(None, feed)[0]

cases = [
("ReduceMax", np.array([-np.inf, -np.inf], dtype=np.float32)),
("ReduceMin", np.array([ np.inf, np.inf], dtype=np.float32)),
]

for op, x in cases:
model = make_model(op)
feed = {"x": x}

cpu_providers, cpu_out = run(
model,
feed,
["CPUExecutionProvider"],
ort.GraphOptimizationLevel.ORT_DISABLE_ALL,
)
cuda_providers, cuda_out = run(
model,
feed,
["CUDAExecutionProvider", "CPUExecutionProvider"],
ort.GraphOptimizationLevel.ORT_ENABLE_ALL,
)
trt_providers, trt_out = run(
model,
feed,
["TensorrtExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"],
ort.GraphOptimizationLevel.ORT_ENABLE_ALL,
)

print(op)
print(" CPU providers:", cpu_providers, "output:", cpu_out)
print(" CUDA providers:", cuda_providers, "output:", cuda_out)
print(" TensorRT providers:", trt_providers, "output:", trt_out)
```

Observed output with onnxruntime-gpu 1.17.1:

```text
ReduceMax
CPU providers: ['CPUExecutionProvider'] output: -inf
CUDA providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'] output: -3.4028235e+38
TensorRT providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] output: -inf
ReduceMin
CPU providers: ['CPUExecutionProvider'] output: inf
CUDA providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'] output: 3.4028235e+38
TensorRT providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] output: inf
```

I repeated the minimal repro several times and got the same result. I also reproduced the same CPU/CUDA mismatch with onnxruntime-gpu 1.18.1.

### Urgency

Not urgent, but this is a numerical correctness issue for CUDAExecutionProvider on IEEE infinity edge cases.

### Platform

Linux

### OS Version

Ubuntu 20.04.5 LTS (x86_64)

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

onnxruntime-gpu 1.17.1 and 1.18.1

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

CUDA

### Execution Provider Library Version

CUDA 11.8, cuDNN 8.6.0, NVIDIA driver 580.105.08, NVIDIA GeForce RTX 3080 Ti

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Python reproduction with CUDAExecutionProvider and compare the ReduceMax and ReduceMin outputs against CPUExecutionProvider. Trace those reduction entry points through the CUDA execution provider's reduction path. Done means the all-infinite cases return IEEE -inf and +inf consistently, with regression coverage for both operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.