microsoft / microsoft/WindowsML

[Bug]: MIGraphX EP crashes in bundled amdhip64_7.dll (access violation, 0xc0000005) compiling a large model on RX 7900 XTX

Open
#15 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
No language data
Stars
29
Forks
5
Avg merge
11h 1m
Merged PRs (30d)
2

Description

Which sample?

N/A — this is a Windows ML runtime crash in the MIGraphX execution provider itself (package MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8, EP version 1.8.57.0), not tied to any sample code in this repo. Filing here per the maintainers' guidance in case triage wants to redirect to microsoft/WindowsAppSDK; happy to move it there if preferred.

Steps to reproduce
  1. On a Windows machine whose only GPU is an AMD Radeon RX 7900 XTX (gfx1100), build a C++
    application using the ONNX Runtime C++ API (via Windows ML's EP catalog) that:
    • Discovers and registers the MIGraphX EP via WinMLEpCatalog (EnsureReady + GetLibraryPath),
      then Ort::Env::RegisterExecutionProviderLibrary("MIGraphX", ).
    • Selects the MIGraphX device via Ort::Env::GetEpDevices(), filtering for EP name
      "MIGraphX", and appends it with SessionOptions::AppendExecutionProvider_V2.
    • Before session creation, pins the ONNX graph's "batch" free dimension to a fixed value
      (e.g. 8) via SessionOptions::AddFreeDimensionOverrideByName("batch", 8), on a ~930MB
      ONNX graph (converted from a KataGo Go-playing neural network; the exact model isn't
      important, any moderately large graph with a pinned static batch dimension should do).
  2. Construct the Ort::Session (first-time compile, no on-disk compile cache present yet for
    this model/shape).
  3. Observe: CPU spikes to 100% for several minutes, host RAM climbs to several GB, VRAM usage
    stays at 0% throughout, then the process crashes with no exception and no error output.

This reproduced identically twice, from two separate process launches (once via an interactive
stdio/GTP-protocol subprocess, once via a one-shot batch-mode subcommand) — same crash, same
module, same offset both times.

Minimal repro code is not attached since this was found via an existing open-source project
(a fork of KataGo with a Windows ML backend); happy to extract a minimal standalone repro if
that would help triage — let me know.

Expected behavior

The model compiles successfully and inference proceeds, OR the failure surfaces as a catchable
ONNX Runtime exception (e.g. Ort::Exception / EP_FAIL) rather than a native access violation.

For reference, the identical model/code/EP-package-version compiles successfully (in ~9 minutes)
on a different machine with an integrated AMD GPU + NPU, so this appears specific to this
discrete-GPU (RX 7900 XTX) configuration.

Actual behavior

The process hard-crashes with an unhandled access violation. Windows Event Viewer (Application
log, Event ID 1000, source "Application Error") shows:

Faulting application name: katago.exe, version: 0.0.0.0, time stamp: 0x6a4fb5f2
Faulting module name: amdhip64_7.dll, version: 7.2.2606.20, time stamp: 0x69e9440d
Exception code: 0xc0000005
Fault offset: 0x0000000000465007
Faulting module path: C:\Program Files\WindowsApps
MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_1.8.57.0_x64__8wekyb3d8bbwe
ExecutionProvider\amdhip64_7.dll

Reproduced twice (different PIDs, different process launches) with the IDENTICAL faulting
offset both times — this rules out a random memory-pressure or race-condition explanation.
System RAM was 64GB with 55%+ free at time of testing, so this is not a simple OOM.

No exception is thrown into the calling application before the crash — the crash happens
entirely inside the EP's own bundled amdhip64_7.dll, downstream of the Ort::Session
constructor, so it cannot be caught by the caller's try/catch around session creation.

Windows version

Windows 11 25H2 OS Internal version: 26200.8737 ( Win32NT 10.0.26200.0 Microsoft Windows NT 10.0.26200.0)

Architecture

x64

Hardware (CPU / GPU / NPU)

Intel Core i7-9700, AMD Radeon RX 7900XTX

Execution provider

MIGraphX (explicitly selected via winmlProvider config option in the calling application; not using automatic EP-selection policy). EP name as reported by the catalog: MIGraphXExecutionProvider. Package: MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_8wekyb3d8bbwe, version 1.8.57.0.

Windows ML / Windows App SDK package version

Microsoft.Windows.AI.MachineLearning 2.0.297-preview

Diagnostic logs (optional)

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported crash on Windows with an RX 7900 XTX, using the WinMLEpCatalog setup and Ort::Session construction described in the issue. Compare the MIGraphX path with the integrated AMD GPU/NPU result and inspect the bundled amdhip64_7.dll fault details. Done means identifying a reproducible cause and a clear fix or escalation target; no repository test is named.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, machine-learning
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.