microsoft / microsoft/onnxruntime
CUDA EP ReduceMax/ReduceMin return finite limits instead of +/-inf for all-infinite inputs
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
### Describe the issue
For floating-point reductions, CUDAExecutionProvider returns finite float limits when all reduced values are infinite, while CPUExecutionProvider and TensorRTExecutionProvider return IEEE +/-inf.
Minimal cases:
- `ReduceMax([-inf, -inf])`: CPU EP returns `-inf`, TensorRT EP returns `-inf`, but CUDA EP returns `-3.4028235e+38`.
- `ReduceMin([+inf, +inf])`: CPU EP returns `+inf`, TensorRT EP returns `+inf`, but CUDA EP returns `+3.4028235e+38`.
This looks like a numerical correctness issue in the CUDA reduction path for valid IEEE floating-point inputs. The result is stable across repeated runs.
I reproduced the issue with:
- onnxruntime-gpu 1.17.1
- onnxruntime-gpu 1.18.1
- Python API
- Linux x86_64
- CUDA 11.8 / cuDNN 8.6
- NVIDIA GeForce RTX 3080 Ti
I also tried checking onnxruntime-gpu 1.23.2, but that wheel requires CUDA 12 / cuDNN 9 and falls back to CPU on this machine, so I could not validate the latest CUDA EP behavior here.
### To reproduce
### To reproduce
Run the following script on a machine with CUDAExecutionProvider available:
```python
import numpy as np
import onnx
import onnxruntime as ort
from onnx import TensorProto, helper
print("onnxruntime", ort.__version__)
print("available providers", ort.get_available_providers())
def make_model(op):
model = helper.make_model(
helper.make_graph(
[helper.make_node(op, ["x"], ["z"], axes=[0], keepdims=0)],
op,
[helper.make_tensor_value_info("x", TensorProto.FLOAT, [2])],
[helper.make_tensor_value_info("z", TensorProto.FLOAT, [])],
),
opset_imports=[helper.make_operatorsetid("", 17)],
)
model.ir_version = 8
onnx.checker.check_model(model)
return model
def run(model, feed, providers, opt):
so = ort.SessionOptions()
so.graph_optimization_level = opt
sess = ort.InferenceSession(model.SerializeToString(), so, providers=providers)
return sess.get_providers(), sess.run(None, feed)[0]
cases = [
("ReduceMax", np.array([-np.inf, -np.inf], dtype=np.float32)),
("ReduceMin", np.array([ np.inf, np.inf], dtype=np.float32)),
]
for op, x in cases:
model = make_model(op)
feed = {"x": x}
cpu_providers, cpu_out = run(
model,
feed,
["CPUExecutionProvider"],
ort.GraphOptimizationLevel.ORT_DISABLE_ALL,
)
cuda_providers, cuda_out = run(
model,
feed,
["CUDAExecutionProvider", "CPUExecutionProvider"],
ort.GraphOptimizationLevel.ORT_ENABLE_ALL,
)
trt_providers, trt_out = run(
model,
feed,
["TensorrtExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"],
ort.GraphOptimizationLevel.ORT_ENABLE_ALL,
)
print(op)
print(" CPU providers:", cpu_providers, "output:", cpu_out)
print(" CUDA providers:", cuda_providers, "output:", cuda_out)
print(" TensorRT providers:", trt_providers, "output:", trt_out)
```
Observed output with onnxruntime-gpu 1.17.1:
```text
ReduceMax
CPU providers: ['CPUExecutionProvider'] output: -inf
CUDA providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'] output: -3.4028235e+38
TensorRT providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] output: -inf
ReduceMin
CPU providers: ['CPUExecutionProvider'] output: inf
CUDA providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'] output: 3.4028235e+38
TensorRT providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] output: inf
```
I repeated the minimal repro several times and got the same result. I also reproduced the same CPU/CUDA mismatch with onnxruntime-gpu 1.18.1.
### Urgency
Not urgent, but this is a numerical correctness issue for CUDAExecutionProvider on IEEE infinity edge cases.
### Platform
Linux
### OS Version
Ubuntu 20.04.5 LTS (x86_64)
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
onnxruntime-gpu 1.17.1 and 1.18.1
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
CUDA
### Execution Provider Library Version
CUDA 11.8, cuDNN 8.6.0, NVIDIA driver 580.105.08, NVIDIA GeForce RTX 3080 Ti
Contributor guide
Research direction
Start by running the supplied Python reproduction with CUDAExecutionProvider and compare the ReduceMax and ReduceMin outputs against CPUExecutionProvider. Trace those reduction entry points through the CUDA execution provider's reduction path. Done means the all-infinite cases return IEEE -inf and +inf consistently, with regression coverage for both operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100