microsoft / microsoft/onnxruntime

[MIGraphX EP] Fails processing models with multiple layers of sub-graphs with "Unknown operator: MGXKernel_subgraph_tf2onnx_9255412890226604894_0"

Open
#31,713 7 comments 0 reactions 0 assignees View on GitHub
ep:MIGraphX stale
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

This is a new issue on top of https://github.com/microsoft/onnxruntime/issues/31607 and the fix for that one (CC @liminfei-amd).

Same model, available from https://gitlab.collabora.com/gstreamer/onnx-models/-/blob/a8de628ad5a745d3af6ba8c0ee84321605de07e9/models/ssd_mobilenet_v1_coco.onnx

Loading this model works fine after the fix for the other issue but doing actual inference fails:

```
./migraphx_sample migraphx onnx-models/models/ssd_mobilenet_v1_coco.onnx
Loading: onnx-models/models/ssd_mobilenet_v1_coco.onnx (provider: migraphx, device: 0)
2026-08-07 10:17:38.429816144 [W:onnxruntime:sample, migraphx_execution_provider.cc:167 MIGraphXExecutionProvider] [MIGraphX EP] MIGraphX ENV Override Variables Set:
Session created successfully.
Input 0: image_tensor:0 shape=[1, 300, 300, 3]
Running inference...
2026-08-07 10:17:38.715384 [WARN] [AMDMIGraphX/src/onnx/parse_loop.cpp:66] PARSE_LOOP max_iterations exceeds the maximum loop iterations limit, it will be changed from 9223372036854775807 to 65535.
Warning: onnx model is not topologically sorted. Attempting to sort...
migraphx_parse_onnx_buffer: Error: AMDMIGraphX/src/onnx/onnx_parser.cpp:681: parse_graph: Unknown operator: MGXKernel_subgraph_tf2onnx_9255412890226604894_0
2026-08-07 10:17:38.717874898 [E:onnxruntime:, sequential_executor.cc:671 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_tf2onnx_9255412890226604894_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_tf2onnx_9255412890226604894_5_5' Status Message: Failed to call function
terminate called after throwing an instance of 'Ort::Exception'
what(): Non-zero status code returned while running MGXKernel_graph_tf2onnx_9255412890226604894_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_tf2onnx_9255412890226604894_5_5' Status Message: Failed to call function
fish: Job 1, './migraphx_sample migraphx /hom…' terminated by signal SIGABRT (Abort)
```

AFAIU the problem here is that the execution provider is partitioning the model recursively and modifies them along the way, which then causes failures because it's working bottom-up and the upper graphs see modified sub-graphs that they can't handle anymore.

With the CPU execution provider the above works.

### To reproduce

Extended minimal sample from the previous issue: [migraphx_sample.cc.txt](https://github.com/user-attachments/files/30819806/migraphx_sample.cc.txt)

Compile and run with the above command and it will fail.

### Urgency

_No response_

### Platform

Linux

### OS Version

Fedora 44

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

1.28.0

### ONNX Runtime API

C++

### Architecture

X64

### Execution Provider

MIGraphX

### Execution Provider Library Version

ROCm 7.15.0 (20260803-30774164023) / MIGraphX rocm-7.14 tag

Contributor guide

Open the contributing guide

Research direction

Start with the attached migraphx_sample.cc reproduction and the MIGraphX execution-provider path around migraphx_execution_provider.cc:167. Trace the inference failure through onnx_parser.cc:681 and the reported parse_graph error, comparing behavior with the CPU execution provider. Done means the supplied ssd_mobilenet_v1_coco.onnx model completes inference through MIGraphX without the unknown subgraph operator error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.