microsoft / microsoft/WindowsAppSDK

VitisAI EP (v1.8.63.0) accepts a valid amd-quark INT8 model but assigns all/some nodes to CPU with zero diagnostic detail

Open
#6,610 0 comments 0 reactions 0 assignees View on GitHub
area-WinML needs-triage
Dominant language
C++
Stars
4.7k
Forks
471
Avg merge
3d 13h
Merged PRs (30d)
28

Description

### Describe the bug

A model quantized to INT8 QDQ format via AMD's own recommended API (`quark.onnx.ModelQuantizer`
with the `XINT8_QCONFIG` preset — the documented power-of-2-scale scheme for Ryzen AI NPU CNN
deployment) loads and runs successfully through the Windows ML VitisAI execution provider, but
verified via `xrt-smi` that none of the computation actually dispatches to the NPU — it silently
runs entirely on CPU. Enabling ONNX Runtime's `session.disable_cpu_ep_fallback=1` confirms this by
turning the silent fallback into a session-creation failure, but that error is session-level only
("this session contains CPU EP nodes") with no per-node detail about which op(s) VitisAI rejected
or why, and neither ORT's `SetLogSeverityLevel(VERBOSE)` nor VitisAI's own apparent glog-based
internal logging (`GLOG_minloglevel=0`, `GLOG_v=3`) surface any additional information.

### Steps to reproduce the bug

1. Quantize a standard FP32 ONNX CNN (in our case: ~40 residual conv blocks, ops limited to
Conv/Add/Mul/Softplus/Tanh/Div/ReduceSum/ReduceMax/Concat — all standard, widely-supported
types) using amd-quark 0.11.2 (`pip install "amd-quark==0.11.2" --extra-index-url
https://pypi.amd.com/quark/cpu/simple`):
from quark.onnx import ModelQuantizer, XINT8_QCONFIG
ModelQuantizer(XINT8_QCONFIG).quantize_model(
model_input=fp32_path, model_output=int8_path, calibration_data_reader=reader)
This completes successfully; the quantizer's own summary table reports all major ops (Conv,
Add, Mul, Softplus, Tanh, etc.) converted to INT8.
2. Load the resulting model through Windows ML, selecting the VitisAI EP device:
auto devices = ort_env.GetEpDevices(); // filter for EpName() == "VitisAIExecutionProvider"
session_options.AppendExecutionProvider_V2(ort_env, vitisai_devices, {});
3. Run inference, and separately monitor `xrt-smi examine -r aie-partitions` for the duration.
4. Also try adding `session_options.AddConfigEntry("session.disable_cpu_ep_fallback", "1")` before
session creation and observe the session-creation-time failure instead.

### Expected behavior

Either the quantized ops actually dispatch to the NPU, or the EP clearly reports (ideally with
per-node detail) which ops it cannot accelerate, instead of two failure modes that are both hard
to diagnose:
(a) with CPU fallback allowed (default): looks fully successful (loads, runs, numerically valid
output) while silently running 100% on CPU;
(b) with CPU fallback disabled: a session-level error with no indication of which node(s) or op
type(s) caused it.

### Screenshots

_No response_

### NuGet package version

Microsoft.Windows.AI.MachineLearning 2.0.297-preview (This is a native C++ project using the Windows ML EP catalog directly via WinMLEpCatalog.h / onnxruntime_cxx_api.h, not a .NET project consuming NuGet in the usual csproj sense — this is the package version CMake fetched. VitisAI EP package itself: MicrosoftCorporationII.WinML.AMD.NPU.EP.1.8, EP version 1.8.63.0, EP name "VitisAIExecutionProvider".)

### Packaging type

Unpackaged (Win32, unpackaged desktop app). Uses the Dynamic Dependencies API at startup to add
Store-distributed EP packages to the process's package graph, per Windows ML's documented pattern
for unpackaged apps.

### Windows version

Windows 11 25H2 OS Insider build: 26200.8737 ( Win32NT 10.0.26200.0 Microsoft Windows NT 10.0.26200.0)

### IDE

Command-line build via CMake + MSVC Build Tools (Visual Studio 18 2026 generator), no IDE
involved in reproducing this issue.

### Additional context

Hardware: AMD Ryzen AI 9 H 365 w/ Radeon 880M ("Strix"). NPU driver 32.0.20102.3930, NPU firmware
1.1.2.64, XRT 2.21.0 (all via `xrt-smi examine`).

Full evidence trail:
1. `xrt-smi examine -r aie-partitions` run continuously during a ~20 second sustained inference
workload (KataGo GTP `kata-analyze`) shows zero HW contexts owned by the inferencing process.
All existing HW contexts on the NPU belong to unrelated Windows system processes
(WorkloadsSessionHost.exe, a Copilot+ background service) — confirming zero NPU utilization
despite "successful" session creation and numerically valid inference output.
2. Setting `session.disable_cpu_ep_fallback=1` turns this into:
"This session contains graph nodes that are assigned to the default CPU EP, but fallback to
CPU EP has been explicitly disabled by the user."
confirming VitisAI genuinely does not claim at least some nodes in this graph, rather than this
being a monitoring/measurement artifact.
3. Attempted to get more detail via `SessionOptions::SetLogSeverityLevel(ORT_LOGGING_LEVEL_VERBOSE)`
and via `GLOG_minloglevel=0` / `GLOG_v=3` environment variables (guessing VitisAI's internal
`vaip_core` compiler uses glog, based on the format of an unrelated crash log seen elsewhere:
glog-style "F :]" FATAL messages referencing
`vaip_core::node_arg_get_shape_i64`, `vaip_core::IPass::run_passes`,
`vaip_core::DpuSubgraphEntryProto::mutable_try_fuse`, etc.) — neither produced any additional
diagnostic output.

Questions:
1. What is the correct/recommended amd-quark quantization configuration to get actual NPU offload
for a standard CNN on this driver/EP version? Is `XINT8_QCONFIG` still the right preset for
this EP build, or does something else need to change to match its supported-op list?
2. Is there any supported way to get per-node EP-assignment diagnostics from the VitisAI EP (which
nodes/ops were claimed vs. rejected, and why)?

(Separately, and possibly related: this same EP catalog stack was also observed to hard-crash with
an access violation inside a bundled `amdhip64_7.dll` on a discrete RX 7900 XTX GPU via the
MIGraphX EP package, reported previously as a distinct issue — flagging in case there's a shared
root cause across these AMD-provided EP packages worth investigating together.)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Windows ML VitisAI EP session with the amd-quark XINT8_QCONFIG model, comparing normal CPU fallback with session.disable_cpu_ep_fallback=1 and monitoring xrt-smi examine -r aie-partitions. Review the reported EP and ONNX Runtime logging controls, then determine whether the supported outcome is a corrected quantization configuration or per-node assignment diagnostics. Done means the unsupported nodes or configuration mismatch are identified and the silent fallback is explainable.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
desktop, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.