[Issue][regression]: MIGraphX 2.17 compiles 4x to 11x slower than 2.16 with no inference gain for YOLO models
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 333
- Forks
- 150
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 54
Description
Problem Description
Compiling YOLO ONNX models with MIGraphX 2.17 takes far longer than with 2.16.
Compile time went from about 1.5 minutes to about 17 minutes, and inference did not get faster.
gfx1201 (R9700 AI PRO), Python 3.12. Each run gets an empty AMD_COMGR_CACHE_DIR and HOME, so every compile is cold. Numbers are medians of 3 runs for 2.16 and single runs for 2.17. ROCm comes bundled with each MIGraphX version, 7.14.0 for 2.16 and 10.0.0 for 2.17.
Compile time, seconds, cold cache
| Model | Batch | 2.16 | 2.17 | Slowdown |
|---|---|---|---|---|
| yolo26n | 1 | 91.9 | 1014.1 | 11.0x |
| yolo26n | 32 | 121.8 | 991.4 | 8.1x |
| yolo26l | 1 | 92.5 | 976.6 | 10.6x |
| yolo26l | 32 | 255.0 | 1115.4 | 4.4x |
Inference, median ms
| Model | Batch | 2.16 | 2.17 | Change |
|---|---|---|---|---|
| yolo26n | 1 | 1.609 | 1.556 | -3.3% |
| yolo26n | 32 | 19.997 | 19.265 | -3.7% |
| yolo26l | 1 | 8.860 | 8.651 | -2.4% |
| yolo26l | 32 | 174.126 | 181.699 | +4.3% |
Every case stays within 4%, so the extra 15 minutes of compilation buys no measurable speedup.
MIGRAPHX_SKIP_BENCHMARKING=1 cuts 2.17 compile to 69 to 94 seconds, but inference then drops by 4 to 27%, so it is not a usable workaround.
Any workaround for 2.17?
Operating System
Ubuntu 24.04.3 LTS
CPU
AMD Ryzen Threadripper PRO 7975WX
GPU
R9700 AI PRO
Other
No response
ROCm Version
ROCm 10
Steps to Reproduce
pip install ultralytics
for m in yolo26n yolo26l; do for b in 1 32; do
yolo export model=$m.pt format=onnx batch=$b imgsz=640 device=cpu simplify=True
mv $m.onnx ${m}_b$b.onnx
done; done
migraphx 2.16
python3.12 -m venv .venv-216 && . .venv-216/bin/activate
pip install "migraphx==2.16.0+rocm7.14.0" --extra-index-url https://rocm.frameworks.amd.com/whl-multi-arch/
pip install "rocm[libraries,device-gfx1201]==7.14.0" --extra-index-url https://repo.amd.com/rocm/whl-multi-arch/
export LD_LIBRARY_PATH=$(find $VIRTUAL_ENV/lib/python3.12/site-packages -name '*.so*' -printf '%h\n' | sort -u | paste -sd:)
migraphx-driver perf --onnx yolo26n_b1.onnx --gpu -n 300
migraphx 2.17
python3.12 -m venv .venv-217 && . .venv-217/bin/activate
pip install "migraphx==2.17.0+rocm10.0.0" --extra-index-url https://stable.repo.amd.com/rocm/migraphx/whl-next/
pip install "rocm[libraries,device-gfx1201]==10.0.0" --extra-index-url https://stable.repo.amd.com/rocm/whl-next/
export LD_LIBRARY_PATH=$(find $VIRTUAL_ENV/lib/python3.12/site-packages -name '*.so*' -printf '%h\n' | sort -u | paste -sd:)
migraphx-driver perf --onnx yolo26n_b1.onnx --gpu -n 300
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided YOLO export and migraphx-driver perf commands against MIGraphX 2.16 and 2.17 with empty AMD_COMGR_CACHE_DIR and HOME. Compare cold-cache compile and inference timings, then trace the regression across the two versions; done means identifying a reproducible cause or workaround that avoids the large compile-time increase without the reported inference loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100