ROCm / ROCm/AMDMIGraphX

[Issue][regression]: MIGraphX 2.17 compiles 4x to 11x slower than 2.16 with no inference gain for YOLO models

Open
#5,234 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
333
Forks
150
Avg merge
4d 19h
Merged PRs (30d)
54

Description

Problem Description

Compiling YOLO ONNX models with MIGraphX 2.17 takes far longer than with 2.16.
Compile time went from about 1.5 minutes to about 17 minutes, and inference did not get faster.

gfx1201 (R9700 AI PRO), Python 3.12. Each run gets an empty AMD_COMGR_CACHE_DIR and HOME, so every compile is cold. Numbers are medians of 3 runs for 2.16 and single runs for 2.17. ROCm comes bundled with each MIGraphX version, 7.14.0 for 2.16 and 10.0.0 for 2.17.

Compile time, seconds, cold cache
Model Batch 2.16 2.17 Slowdown
yolo26n 1 91.9 1014.1 11.0x
yolo26n 32 121.8 991.4 8.1x
yolo26l 1 92.5 976.6 10.6x
yolo26l 32 255.0 1115.4 4.4x
Inference, median ms
Model Batch 2.16 2.17 Change
yolo26n 1 1.609 1.556 -3.3%
yolo26n 32 19.997 19.265 -3.7%
yolo26l 1 8.860 8.651 -2.4%
yolo26l 32 174.126 181.699 +4.3%

Every case stays within 4%, so the extra 15 minutes of compilation buys no measurable speedup.

MIGRAPHX_SKIP_BENCHMARKING=1 cuts 2.17 compile to 69 to 94 seconds, but inference then drops by 4 to 27%, so it is not a usable workaround.

Any workaround for 2.17?

Operating System

Ubuntu 24.04.3 LTS

CPU

AMD Ryzen Threadripper PRO 7975WX

GPU

R9700 AI PRO

Other

No response

ROCm Version

ROCm 10

Steps to Reproduce
pip install ultralytics
for m in yolo26n yolo26l; do for b in 1 32; do
  yolo export model=$m.pt format=onnx batch=$b imgsz=640 device=cpu simplify=True
  mv $m.onnx ${m}_b$b.onnx
done; done

migraphx 2.16

python3.12 -m venv .venv-216 && . .venv-216/bin/activate
pip install "migraphx==2.16.0+rocm7.14.0" --extra-index-url https://rocm.frameworks.amd.com/whl-multi-arch/
pip install "rocm[libraries,device-gfx1201]==7.14.0" --extra-index-url https://repo.amd.com/rocm/whl-multi-arch/
export LD_LIBRARY_PATH=$(find $VIRTUAL_ENV/lib/python3.12/site-packages -name '*.so*' -printf '%h\n' | sort -u | paste -sd:)
migraphx-driver perf --onnx yolo26n_b1.onnx --gpu -n 300

migraphx 2.17

python3.12 -m venv .venv-217 && . .venv-217/bin/activate
pip install "migraphx==2.17.0+rocm10.0.0" --extra-index-url https://stable.repo.amd.com/rocm/migraphx/whl-next/
pip install "rocm[libraries,device-gfx1201]==10.0.0" --extra-index-url https://stable.repo.amd.com/rocm/whl-next/
export LD_LIBRARY_PATH=$(find $VIRTUAL_ENV/lib/python3.12/site-packages -name '*.so*' -printf '%h\n' | sort -u | paste -sd:)
migraphx-driver perf --onnx yolo26n_b1.onnx --gpu -n 300
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided YOLO export and migraphx-driver perf commands against MIGraphX 2.16 and 2.17 with empty AMD_COMGR_CACHE_DIR and HOME. Compare cold-cache compile and inference timings, then trace the regression across the two versions; done means identifying a reproducible cause or workaround that avoids the large compile-time increase without the reported inference loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.