microsoft / microsoft/onnxruntime

[MIGraphX EP] Models with 10+ outputs fail with "invalid input for conversion to integer": output name "main:#output_:00010" is not parsed

Open
#32,567 0 comments 0 reactions 0 assignees View on GitHub
ep:MIGraphX
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

With current MIGraphX (`develop`), any model with **10 or more outputs** fails in the MIGraphX EP with

```
invalid input for conversion to integer
```

### Cause

MIGraphX names program outputs with `param_name()` (`src/param_utils.cpp`, since ROCm/AMDMIGraphX#3054; used for outputs in `src/replace_allocate.cpp`: `param_name(index++, mod.name() + ":#output_")`):

```cpp
if(i < 10)
return prefix + std::to_string(i);
...
return prefix + ":" + std::string(max_digits - n, '0') + std::to_string(i);
```

So outputs are named `main:#output_0` … `main:#output_9`, then `main:#output_:00010`, `main:#output_:00011`, …

The EP extracts the index in `onnxruntime/core/providers/migraphx/migraphx_execution_provider.cc` (`compute_output_index`, ~line 1557):

```cpp
const auto index_str = sv.substr(pos + out_name_prefix.length()); // ":00010"
return ToInteger(Trim(index_str, std::isdigit));
```

For `":00010"`, `Trim(index_str, std::isdigit)` returns `":0001"` (`TrimLeft` in `migraphx_execution_provider_utils.h` returns `sv.substr(0, sv.end() - first_match)`, i.e. it cuts the tail instead of the leading non-digit), and `ToInteger` (`std::from_chars`) throws. Outputs 0–9 have no leading non-digit, so the problem only shows up from the 10th output on.

**Expected:** indices 10, 11, … are parsed and the model runs.

The same code is on `main`, `v1.29.0`, `v1.29.1` and `v1.30.0` (it came with #25583).

### Suggested fix

This is what we apply locally; it fixes the issue (both for the synthetic model below and for a real 12-output face detector):

```cpp
const auto digits = index_str.find_first_of("0123456789");
return digits == std::string_view::npos ? -1 : ToInteger(index_str.substr(digits));
```

(`from_chars` accepts the leading zeros.) `TrimLeft` itself may deserve a separate look, since it keeps the wrong part of the string.

### To reproduce

Any ONNX model with 10+ outputs. Minimal example (12 outputs):

```python
import numpy as np, onnx, onnxruntime as ort
from onnx import TensorProto, helper

n = 12
nodes = [helper.make_node("Mul", ["x", f"c{i}"], [f"y{i}"]) for i in range(n)]
inits = [helper.make_tensor(f"c{i}", TensorProto.FLOAT, [1], [float(i + 1)]) for i in range(n)]
graph = helper.make_graph(nodes, "many_outputs",
[helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 8])],
[helper.make_tensor_value_info(f"y{i}", TensorProto.FLOAT, [1, 8]) for i in range(n)], inits)
onnx.save(helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]), "many_outputs.onnx")

sess = ort.InferenceSession("many_outputs.onnx", providers=["MIGraphXExecutionProvider"])
print([float(o[0, 0]) for o in sess.run(None, {"x": np.ones((1, 8), np.float32)})])
```

MIGraphX gives this model the output names above:

```
$ migraphx-driver compile many_outputs.onnx --gpu | grep -o "main:#output_[:0-9]*" | sort -u
main:#output_:00010
main:#output_:00011
main:#output_1
...
main:#output_9
```

Without the fix we got `invalid input for conversion to integer` on a real model with 12 outputs — the YuNet face detector from opencv_zoo (`face_detection_yunet_2023mar.onnx`). With the fix above, both YuNet and the script above run correctly (the script prints `[1.0, 2.0, …, 12.0]`).

### Urgency

Not blocking for us (we carry the patch above), but every model with 10+ outputs — e.g. detectors with multi-scale heads — cannot run on the MIGraphX EP with current MIGraphX.

### Platform

Linux

### OS Version

Ubuntu 24.04.5 LTS (amdgpu-dkms 6.19.14), GCC 13, Python 3.12.3

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

1.31.0, main @ a7df32cf6087a11884042a2a95526d72100e3b95

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

MIGraphX

### Execution Provider Library Version

MIGraphX develop @ a9782157c (2.18.0.dev), ROCm 7.14.1, gfx1102 (Radeon RX 7600 XT)

Contributor guide

Open the contributing guide

Research direction

Start with compute_output_index in onnxruntime/core/providers/migraphx/migraphx_execution_provider.cc and inspect the related trimming behavior in migraphx_execution_provider_utils.h. Reproduce the failure with the provided 12-output Python model, then verify that indices 10 and above parse correctly and the model returns all expected outputs through the MIGraphX EP.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.