microsoft / microsoft/WindowsAppSDK

MIGraphX EP crashes in bundled amdhip64_7.dll (access violation, 0xc0000005) compiling a large model on RX 7900 XTX

Open
#6,611 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area-WinML needs-triage
Dominant language
C++
Stars
4.7k
Forks
471
Avg merge
3d 13h
Merged PRs (30d)
28

Description

### Describe the bug

This is a Windows ML runtime crash in the MIGraphX execution provider itself (package MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8, EP version 1.8.57.0), not tied to any sample code in this repo. Filing here per the maintainers' guidance in case triage wants to redirect to microsoft/WindowsAppSDK; happy to move it there if preferred.

### Steps to reproduce the bug

1. On a Windows machine whose only GPU is an AMD Radeon RX 7900 XTX (gfx1100), build a C++
application using the ONNX Runtime C++ API (via Windows ML's EP catalog) that:
- Discovers and registers the MIGraphX EP via WinMLEpCatalog (EnsureReady + GetLibraryPath),
then Ort::Env::RegisterExecutionProviderLibrary("MIGraphX", ).
- Selects the MIGraphX device via Ort::Env::GetEpDevices(), filtering for EP name
"MIGraphX", and appends it with SessionOptions::AppendExecutionProvider_V2.
- Before session creation, pins the ONNX graph's "batch" free dimension to a fixed value
(e.g. 8) via SessionOptions::AddFreeDimensionOverrideByName("batch", 8), on a ~930MB
ONNX graph (converted from a KataGo Go-playing neural network; the exact model isn't
important, any moderately large graph with a pinned static batch dimension should do).
2. Construct the Ort::Session (first-time compile, no on-disk compile cache present yet for
this model/shape).
3. Observe: CPU spikes to 100% for several minutes, host RAM climbs to several GB, VRAM usage
stays at 0% throughout, then the process crashes with no exception and no error output.

This reproduced identically twice, from two separate process launches (once via an interactive
stdio/GTP-protocol subprocess, once via a one-shot batch-mode subcommand) — same crash, same
module, same offset both times.

Minimal repro code is not attached since this was found via an existing open-source project
(a fork of KataGo with a Windows ML backend); happy to extract a minimal standalone repro if
that would help triage — let me know.

### Expected behavior

The model compiles successfully and inference proceeds, OR the failure surfaces as a catchable
ONNX Runtime exception (e.g. Ort::Exception / EP_FAIL) rather than a native access violation.

For reference, the identical model/code/EP-package-version compiles successfully (in ~9 minutes)
on a different machine with an integrated AMD GPU + NPU, so this appears specific to this
discrete-GPU (RX 7900 XTX) configuration.

### Screenshots

The process hard-crashes with an unhandled access violation. Windows Event Viewer (Application
log, Event ID 1000, source "Application Error") shows:

Faulting application name: katago.exe, version: 0.0.0.0, time stamp: 0x6a4fb5f2
Faulting module name: amdhip64_7.dll, version: 7.2.2606.20, time stamp: 0x69e9440d
Exception code: 0xc0000005
Fault offset: 0x0000000000465007
Faulting module path: C:\Program Files\WindowsApps\
MicrosoftCorporationII.WinML.AMD.GPU.EP.1.8_1.8.57.0_x64__8wekyb3d8bbwe\
ExecutionProvider\amdhip64_7.dll

Reproduced twice (different PIDs, different process launches) with the IDENTICAL faulting
offset both times — this rules out a random memory-pressure or race-condition explanation.
System RAM was 64GB with 55%+ free at time of testing, so this is not a simple OOM.

No exception is thrown into the calling application before the crash — the crash happens
entirely inside the EP's own bundled amdhip64_7.dll, downstream of the Ort::Session
constructor, so it cannot be caught by the caller's try/catch around session creation.

### NuGet package version

Microsoft.Windows.AI.MachineLearning 2.0.297-preview (This is a native C++ project using the Windows ML EP catalog directly via WinMLEpCatalog.h / onnxruntime_cxx_api.h, not a .NET project consuming NuGet in the usual csproj sense — this is the package version CMake fetched. VitisAI EP package itself: MicrosoftCorporationII.WinML.AMD.NPU.EP.1.8, EP version 1.8.63.0, EP name "VitisAIExecutionProvider".)

### Packaging type

Unpackaged (Win32, unpackaged desktop app). Uses the Dynamic Dependencies API at startup to add
Store-distributed EP packages to the process's package graph, per Windows ML's documented pattern
for unpackaged apps.

### Windows version

Windows 11 25H2 OS Insider build: 26200.8737 ( Win32NT 10.0.26200.0 Microsoft Windows NT 10.0.26200.0)

### IDE

Command-line build via CMake + MSVC Build Tools (Visual Studio 18 2026 generator), no IDE involved in reproducing this issue.

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the WinMLEpCatalog setup and Ort::Session construction described in the report, including AddFreeDimensionOverrideByName and AppendExecutionProvider_V2. Reproduce on the RX 7900 XTX configuration and inspect the amdhip64_7.dll crash details; done means identifying an actionable owner or fix, or confirming that the issue should move to the MIGraphX or Windows ML project.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
desktop, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.