ROCm / ROCm/AMDMIGraphX

[Feature][ONNX][MIGraphX EP] Offline pre-compilation, bundling, and batch-size-aware loading of `.mxr` artifacts for zero-stall deployment

Open
#5,037 5 comments 0 reactions 1 assignee View on GitHub

@TedThemistokleous is already working on this.

Since Jul 22, 2026.

Dominant language
C++
Stars
333
Forks
150
Avg merge
4d 19h
Merged PRs (30d)
54

Description

DOR (Definition of Ready)
  • migraphx-driver compile --onnx --gpu --binary -o already produces valid .mxr (msgpack) output, confirmed compatible with migraphx::load() — the offline compilation primitive already exists
  • ORT MIGraphX EP already loads .mxr from disk via preload_mxr_cache_from_disk() (introduced in commit ROCm/onnxruntime@9c1485648f)
  • Current gap: the EP locates .mxr files by an internal hash (make_hash() at migraphx_execution_provider.cc, MurmurHash3 over input shapes) that external tooling cannot reproduce, and only from the global ORT_MIGRAPHX_MODEL_CACHE_PATH — so there is no externalized, batch-size-addressable contract for pre-built artifacts
  • Deployment stack: Triton Inference Server → ORT backend → MIGraphX EP, on AMD MI300X (gfx942)
Description

As a model serving infra engineer deploying ONNX models on AMD MI300X via Triton + ORT MIGraphX EP,
I want to (1) pre-compile an ONNX model into per-batch-size .mxr artifacts in an offline environment,
(2) bundle and distribute those artifacts alongside the model, and (3) have the Triton ORT-backend
MIGraphX EP recognize and load the correct artifact by batch size at runtime through an externalized,
documented contract — so that inference never pays on-demand compilation stall time on the GPU host.

The offline compilation primitive already exists (migraphx-driver ... --binary), and the EP can
already load .mxr from disk. What is missing is the externalized contract that ties the three stages
together:

  • Offline: a reproducible way to know which .mxr filename (or manifest entry) corresponds to a
    given (model, batch size, precision) — today the mapping is an internal hash that cannot be
    reproduced outside a live InferenceSession.
  • Distribution: a way to ship pre-built, read-only artifacts as part of a model bundle, rather
    than depending on the single global writable ORT_MIGRAPHX_MODEL_CACHE_PATH.
  • Runtime: the EP loads the matching artifact by batch size from that contract, falling back to
    on-demand compilation only on a miss.
DOD (Definition of Done)

Core requirement — an externalized, batch-size-addressable artifact contract:

  • A reproducible, documented mapping from (model, batch size, precision flags) to its .mxr
    artifact — via a manifest file, a documented filename formula, or a CLI/Python helper — so
    offline-built artifacts can be placed where the EP will find them without a live session
  • The Triton ORT-backend MIGraphX EP loads the matching .mxr by batch size from this contract
    at session init; on a miss it falls back to on-demand compilation and writes to
    ORT_MIGRAPHX_MODEL_CACHE_PATH as today
  • End-to-end verified on MI300X: offline compile per batch size → bundle + distribute →
    zero first-inference stall under Triton

Potential implementation (non-binding):

  • A per-model precompiled artifact path, e.g. a migraphx_precompiled_path provider option passed
    through the parameters block of gpu_execution_accelerator in Triton's config.pbtxt,
    resolved relative to the model version directory (./mxrmodel_repository/<model>/1/mxr/),
    keeping each model bundle self-contained and read-only-friendly

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.