microsoft / microsoft/onnxruntime
[MIGraphX EP] Models with 10+ outputs fail with "invalid input for conversion to integer": output name "main:#output_:00010" is not parsed
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
With current MIGraphX (`develop`), any model with **10 or more outputs** fails in the MIGraphX EP with
```
invalid input for conversion to integer
```
### Cause
MIGraphX names program outputs with `param_name()` (`src/param_utils.cpp`, since ROCm/AMDMIGraphX#3054; used for outputs in `src/replace_allocate.cpp`: `param_name(index++, mod.name() + ":#output_")`):
```cpp
if(i < 10)
return prefix + std::to_string(i);
...
return prefix + ":" + std::string(max_digits - n, '0') + std::to_string(i);
```
So outputs are named `main:#output_0` … `main:#output_9`, then `main:#output_:00010`, `main:#output_:00011`, …
The EP extracts the index in `onnxruntime/core/providers/migraphx/migraphx_execution_provider.cc` (`compute_output_index`, ~line 1557):
```cpp
const auto index_str = sv.substr(pos + out_name_prefix.length()); // ":00010"
return ToInteger(Trim(index_str, std::isdigit));
```
For `":00010"`, `Trim(index_str, std::isdigit)` returns `":0001"` (`TrimLeft` in `migraphx_execution_provider_utils.h` returns `sv.substr(0, sv.end() - first_match)`, i.e. it cuts the tail instead of the leading non-digit), and `ToInteger` (`std::from_chars`) throws. Outputs 0–9 have no leading non-digit, so the problem only shows up from the 10th output on.
**Expected:** indices 10, 11, … are parsed and the model runs.
The same code is on `main`, `v1.29.0`, `v1.29.1` and `v1.30.0` (it came with #25583).
### Suggested fix
This is what we apply locally; it fixes the issue (both for the synthetic model below and for a real 12-output face detector):
```cpp
const auto digits = index_str.find_first_of("0123456789");
return digits == std::string_view::npos ? -1 : ToInteger(index_str.substr(digits));
```
(`from_chars` accepts the leading zeros.) `TrimLeft` itself may deserve a separate look, since it keeps the wrong part of the string.
### To reproduce
Any ONNX model with 10+ outputs. Minimal example (12 outputs):
```python
import numpy as np, onnx, onnxruntime as ort
from onnx import TensorProto, helper
n = 12
nodes = [helper.make_node("Mul", ["x", f"c{i}"], [f"y{i}"]) for i in range(n)]
inits = [helper.make_tensor(f"c{i}", TensorProto.FLOAT, [1], [float(i + 1)]) for i in range(n)]
graph = helper.make_graph(nodes, "many_outputs",
[helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 8])],
[helper.make_tensor_value_info(f"y{i}", TensorProto.FLOAT, [1, 8]) for i in range(n)], inits)
onnx.save(helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)]), "many_outputs.onnx")
sess = ort.InferenceSession("many_outputs.onnx", providers=["MIGraphXExecutionProvider"])
print([float(o[0, 0]) for o in sess.run(None, {"x": np.ones((1, 8), np.float32)})])
```
MIGraphX gives this model the output names above:
```
$ migraphx-driver compile many_outputs.onnx --gpu | grep -o "main:#output_[:0-9]*" | sort -u
main:#output_:00010
main:#output_:00011
main:#output_1
...
main:#output_9
```
Without the fix we got `invalid input for conversion to integer` on a real model with 12 outputs — the YuNet face detector from opencv_zoo (`face_detection_yunet_2023mar.onnx`). With the fix above, both YuNet and the script above run correctly (the script prints `[1.0, 2.0, …, 12.0]`).
### Urgency
Not blocking for us (we carry the patch above), but every model with 10+ outputs — e.g. detectors with multi-scale heads — cannot run on the MIGraphX EP with current MIGraphX.
### Platform
Linux
### OS Version
Ubuntu 24.04.5 LTS (amdgpu-dkms 6.19.14), GCC 13, Python 3.12.3
### ONNX Runtime Installation
Built from Source
### ONNX Runtime Version or Commit ID
1.31.0, main @ a7df32cf6087a11884042a2a95526d72100e3b95
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
MIGraphX
### Execution Provider Library Version
MIGraphX develop @ a9782157c (2.18.0.dev), ROCm 7.14.1, gfx1102 (Radeon RX 7600 XT)
Contributor guide
Research direction
Start with compute_output_index in onnxruntime/core/providers/migraphx/migraphx_execution_provider.cc and inspect the related trimming behavior in migraphx_execution_provider_utils.h. Reproduce the failure with the provided 12-output Python model, then verify that indices 10 and above parse correctly and the model returns all expected outputs through the MIGraphX EP.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100