microsoft / microsoft/onnxruntime

ORT CUDA optimizer's constant-folding pass produces a result that disagrees with the runtime kernel.

Open
#31,140 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ep:CUDA stale
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

The CUDAExecutionProvider's graph optimizer constant-folds `ArgMin` on inputs
containing NaN values. The constant-folded result disagrees with the actual
runtime CUDA kernel, producing wrong indices.

**Computation chain:**
```
Constant(zeros[2,2]) → LogSoftmax → Log → Where(mask, NaN, 0) → ArgMin
```

Since `Log(LogSoftmax(zeros))` = `Log(-log(2))` = NaN, the `Where` node produces
a tensor with a mixture of NaN and 0.0 values. The CUDA optimizer constant-folds
this entire subgraph but computes `ArgMin` using different NaN semantics than the
runtime kernel.

| Input row | Expected (ref, optimizer off) | Actual (opt, optimizer on) |
|-------------------------------|-------------------------------|----------------------------|
| `[NaN, NaN]` → ArgMin | 0 | 0 (agrees) |
| `[0, NaN]` → ArgMin | 1 (index of 0.0, non-NaN min)| **0** (index of NaN) ✗ |

**Effect:** The optimizer returns `[0, 0]` instead of `[0, 1]`.

**Important:** This only reproduces with `CUDAExecutionProvider`. The CPU
optimizer handles NaN correctly.

### To reproduce

```python
from onnx import helper, TensorProto
import onnxruntime as ort
import numpy as np

input_t = helper.make_tensor("v29", TensorProto.FLOAT, [2, 2],
[0.0, 0.0, 0.0, 0.0])
mask_t = helper.make_tensor("mk", TensorProto.BOOL, [2, 2],
[1, 1, 0, 1])
z_t = helper.make_tensor("z", TensorProto.FLOAT, [], [0.0])

nodes = [
helper.make_node("Constant", inputs=[], outputs=["v29"], value=input_t),
helper.make_node("LogSoftmax", inputs=["v29"], outputs=["v34"], axis=-1),
helper.make_node("Log", inputs=["v34"], outputs=["v39"]),
helper.make_node("Constant", inputs=[], outputs=["mk"], value=mask_t),
helper.make_node("Constant", inputs=[], outputs=["z"], value=z_t),
helper.make_node("Where", inputs=["mk", "v39", "z"], outputs=["v44"]),
helper.make_node("ArgMin", inputs=["v44"], outputs=["result"], axis=-1, keepdims=0),
]

out_vi = helper.make_tensor_value_info("result", TensorProto.INT64, [2])
graph = helper.make_graph(nodes, "minimal", [], [out_vi])
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 11)])

# With CUDA optimizer (default)
sess_opt = ort.InferenceSession(
model.SerializeToString(),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

# With CUDA optimizer disabled (reference)
so_ref = ort.SessionOptions()
so_ref.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
sess_ref = ort.InferenceSession(
model.SerializeToString(),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
sess_options=so_ref,
)

opt = np.asarray(sess_opt.run(None, {})[0])
ref = np.asarray(sess_ref.run(None, {})[0])

print(f"opt (optimizer on): {opt}") # [0 0]
print(f"ref (optimizer off): {ref}") # [0 1]
assert not np.allclose(opt, ref), "BUG DID NOT REPRODUCE"
```

### Urgency

No. I found this bug by fuzzing testing. But the bug seems to have medium severity

### Platform

Linux

### OS Version

Ubuntu 5.4.0-162-generic

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.27.0

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

CUDA

### Execution Provider Library Version

CUDA 13.0, Driver 580.76.05, GPU: NVIDIA RTX 3080 Ti

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python reproduction with CUDAExecutionProvider enabled and disabled, then inspect the CUDAExecutionProvider graph optimizer's constant-folding pass and its ArgMin handling. Done means the optimized result matches the reference result, returning [0, 1] for the reported computation while preserving the all-NaN case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.