microsoft / microsoft/onnxruntime
[MIGraphX EP] Fails processing models with multiple layers of sub-graphs with "Unknown operator: MGXKernel_subgraph_tf2onnx_9255412890226604894_0"
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
This is a new issue on top of https://github.com/microsoft/onnxruntime/issues/31607 and the fix for that one (CC @liminfei-amd).
Same model, available from https://gitlab.collabora.com/gstreamer/onnx-models/-/blob/a8de628ad5a745d3af6ba8c0ee84321605de07e9/models/ssd_mobilenet_v1_coco.onnx
Loading this model works fine after the fix for the other issue but doing actual inference fails:
```
./migraphx_sample migraphx onnx-models/models/ssd_mobilenet_v1_coco.onnx
Loading: onnx-models/models/ssd_mobilenet_v1_coco.onnx (provider: migraphx, device: 0)
2026-08-07 10:17:38.429816144 [W:onnxruntime:sample, migraphx_execution_provider.cc:167 MIGraphXExecutionProvider] [MIGraphX EP] MIGraphX ENV Override Variables Set:
Session created successfully.
Input 0: image_tensor:0 shape=[1, 300, 300, 3]
Running inference...
2026-08-07 10:17:38.715384 [WARN] [AMDMIGraphX/src/onnx/parse_loop.cpp:66] PARSE_LOOP max_iterations exceeds the maximum loop iterations limit, it will be changed from 9223372036854775807 to 65535.
Warning: onnx model is not topologically sorted. Attempting to sort...
migraphx_parse_onnx_buffer: Error: AMDMIGraphX/src/onnx/onnx_parser.cpp:681: parse_graph: Unknown operator: MGXKernel_subgraph_tf2onnx_9255412890226604894_0
2026-08-07 10:17:38.717874898 [E:onnxruntime:, sequential_executor.cc:671 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_tf2onnx_9255412890226604894_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_tf2onnx_9255412890226604894_5_5' Status Message: Failed to call function
terminate called after throwing an instance of 'Ort::Exception'
what(): Non-zero status code returned while running MGXKernel_graph_tf2onnx_9255412890226604894_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_tf2onnx_9255412890226604894_5_5' Status Message: Failed to call function
fish: Job 1, './migraphx_sample migraphx /hom…' terminated by signal SIGABRT (Abort)
```
AFAIU the problem here is that the execution provider is partitioning the model recursively and modifies them along the way, which then causes failures because it's working bottom-up and the upper graphs see modified sub-graphs that they can't handle anymore.
With the CPU execution provider the above works.
### To reproduce
Extended minimal sample from the previous issue: [migraphx_sample.cc.txt](https://github.com/user-attachments/files/30819806/migraphx_sample.cc.txt)
Compile and run with the above command and it will fail.
### Urgency
_No response_
### Platform
Linux
### OS Version
Fedora 44
### ONNX Runtime Installation
Built from Source
### ONNX Runtime Version or Commit ID
1.28.0
### ONNX Runtime API
C++
### Architecture
X64
### Execution Provider
MIGraphX
### Execution Provider Library Version
ROCm 7.15.0 (20260803-30774164023) / MIGraphX rocm-7.14 tag
Contributor guide
Research direction
Start with the attached migraphx_sample.cc reproduction and the MIGraphX execution-provider path around migraphx_execution_provider.cc:167. Trace the inference failure through onnx_parser.cc:681 and the reported parse_graph error, comparing behavior with the CPU execution provider. Done means the supplied ssd_mobilenet_v1_coco.onnx model completes inference through MIGraphX without the unknown subgraph operator error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100