microsoft / microsoft/onnxruntime
ORT CUDA optimizer's constant-folding pass produces a result that disagrees with the runtime kernel.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
The CUDAExecutionProvider's graph optimizer constant-folds `ArgMin` on inputs
containing NaN values. The constant-folded result disagrees with the actual
runtime CUDA kernel, producing wrong indices.
**Computation chain:**
```
Constant(zeros[2,2]) → LogSoftmax → Log → Where(mask, NaN, 0) → ArgMin
```
Since `Log(LogSoftmax(zeros))` = `Log(-log(2))` = NaN, the `Where` node produces
a tensor with a mixture of NaN and 0.0 values. The CUDA optimizer constant-folds
this entire subgraph but computes `ArgMin` using different NaN semantics than the
runtime kernel.
| Input row | Expected (ref, optimizer off) | Actual (opt, optimizer on) |
|-------------------------------|-------------------------------|----------------------------|
| `[NaN, NaN]` → ArgMin | 0 | 0 (agrees) |
| `[0, NaN]` → ArgMin | 1 (index of 0.0, non-NaN min)| **0** (index of NaN) ✗ |
**Effect:** The optimizer returns `[0, 0]` instead of `[0, 1]`.
**Important:** This only reproduces with `CUDAExecutionProvider`. The CPU
optimizer handles NaN correctly.
### To reproduce
```python
from onnx import helper, TensorProto
import onnxruntime as ort
import numpy as np
input_t = helper.make_tensor("v29", TensorProto.FLOAT, [2, 2],
[0.0, 0.0, 0.0, 0.0])
mask_t = helper.make_tensor("mk", TensorProto.BOOL, [2, 2],
[1, 1, 0, 1])
z_t = helper.make_tensor("z", TensorProto.FLOAT, [], [0.0])
nodes = [
helper.make_node("Constant", inputs=[], outputs=["v29"], value=input_t),
helper.make_node("LogSoftmax", inputs=["v29"], outputs=["v34"], axis=-1),
helper.make_node("Log", inputs=["v34"], outputs=["v39"]),
helper.make_node("Constant", inputs=[], outputs=["mk"], value=mask_t),
helper.make_node("Constant", inputs=[], outputs=["z"], value=z_t),
helper.make_node("Where", inputs=["mk", "v39", "z"], outputs=["v44"]),
helper.make_node("ArgMin", inputs=["v44"], outputs=["result"], axis=-1, keepdims=0),
]
out_vi = helper.make_tensor_value_info("result", TensorProto.INT64, [2])
graph = helper.make_graph(nodes, "minimal", [], [out_vi])
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 11)])
# With CUDA optimizer (default)
sess_opt = ort.InferenceSession(
model.SerializeToString(),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
# With CUDA optimizer disabled (reference)
so_ref = ort.SessionOptions()
so_ref.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
sess_ref = ort.InferenceSession(
model.SerializeToString(),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
sess_options=so_ref,
)
opt = np.asarray(sess_opt.run(None, {})[0])
ref = np.asarray(sess_ref.run(None, {})[0])
print(f"opt (optimizer on): {opt}") # [0 0]
print(f"ref (optimizer off): {ref}") # [0 1]
assert not np.allclose(opt, ref), "BUG DID NOT REPRODUCE"
```
### Urgency
No. I found this bug by fuzzing testing. But the bug seems to have medium severity
### Platform
Linux
### OS Version
Ubuntu 5.4.0-162-generic
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.27.0
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
CUDA
### Execution Provider Library Version
CUDA 13.0, Driver 580.76.05, GPU: NVIDIA RTX 3080 Ti
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied Python reproduction with CUDAExecutionProvider enabled and disabled, then inspect the CUDAExecutionProvider graph optimizer's constant-folding pass and its ArgMin handling. Done means the optimized result matches the reference result, returning [0, 1] for the reported computation while preserving the all-NaN case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100