microsoft / microsoft/onnxruntime

[OpenVINO][Windows] onnxruntime_provider_test silently exits during MatMulNBits.Float32_4b_Accuracy0

Open
#31,760 1 comment 0 reactions 0 assignees View on GitHub
ep:oneDNN ep:OpenVINO platform:windows
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

## Describe the issue

The Windows OpenVINO CI job [`93078339501`](https://github.com/microsoft/onnxruntime/actions/runs/31247477067/job/93078339501) failed because `onnxruntime_provider_test.exe` terminated silently while running `MatMulNBits.Float32_4b_Accuracy0`.

This was not a logged numerical accuracy failure:

- `MatMulNBits.Float32_4b_Accuracy0` started at `2026-08-08T08:21:30.4228366Z`.
- About 4.1 seconds later, CTest reported `onnxruntime_provider_test` as failed.
- There was no `[ OK ]` or `[ FAILED ]` result for the test, no gtest assertion, no C++ exception, and no Windows exception/status code in the log.
- CTest reported `91% tests passed, 1 tests failed out of 11`, returned exit status 8, and `build.py` consequently returned 1.
- The NPU availability skips and expected negative-test error logs earlier in the job were unrelated.

The retry, job [`93163934771`](https://github.com/microsoft/onnxruntime/actions/runs/31247477067/job/93163934771), passed the same test in 5.117 seconds and completed `onnxruntime_provider_test` successfully.

### Environment comparison

| | Failed attempt | Successful retry |
|---|---|---|
| ORT commit | `6c227c76265711189e15e08e5333430a29a897e7` | Same |
| OpenVINO | `2026.2.1` | Same |
| OpenVINO package commit | `ede283a88e3` | Same |
| CPU | AMD EPYC 9V74 80-Core Processor | Same |
| Runner | `4f589377c000005` | `862958acc000001` |

The observed failure is therefore intermittent and runner-instance-specific so far.

### Source analysis

- The workflow downloads `openvino_toolkit_windows_vc_mt_2026.2.1.21919.ede283a88e3_x86_64.zip`; commit `ede283a88e3` matches the OpenVINO `2026.2.1` tag.
- OpenVINO's `com.microsoft::MatMulNBits` frontend implementation is identical between tags `2026.1.0` and `2026.2.1`. It lowers the op to `u4 Convert -> Subtract -> Multiply -> Reshape/Slice -> MatMul`.
- The relevant OpenVINO CPU conversion/FullyConnected source is also unchanged between those versions. The downstream oneDNN revision does differ (`6b6492b1` to `87f65fdd`), including FP32 matmul dispatch changes, but there is not enough evidence to identify that as the cause.
- `OpTester` runs registered providers separately, CPU first and OpenVINO later. Because the process exited without an exception or provider message, the log does not identify which EP was active at termination.
- No crash dump or test-result artifact was uploaded by the failed run.

PR [#31759](https://github.com/microsoft/onnxruntime/pull/31759) proposes pinning Windows OpenVINO CI to 2026.1.0. That may be a temporary mitigation, but the current evidence does not establish a deterministic OpenVINO 2026.2.1 regression, and the MatMulNBits frontend itself is unchanged between the two versions.

## To reproduce

The failure is not yet reliably reproducible:

1. Build commit `6c227c76265711189e15e08e5333430a29a897e7` on Windows x64 with OpenVINO 2026.2.1 using `.github/workflows/windows_openvino.yml`.
2. Run `onnxruntime_provider_test.exe --gtest_filter=MatMulNBits.Float32_4b_Accuracy0` repeatedly.
3. The first CI attempt terminated during the test; the retry passed on another runner with the same CPU model.

Suggested diagnostics for the next recurrence:

1. Add temporary explicit CPU-only and OpenVINO-only variants to identify the active EP.
2. Run the isolated test with `--gtest_repeat=100`.
3. Enable Windows Error Reporting LocalDumps or ProcDump for `onnxruntime_provider_test.exe`, and upload full dumps as workflow artifacts.
4. Set `ONEDNN_VERBOSE=1` to capture the selected primitive and effective ISA before termination.
5. Record the child process NT exit status rather than only CTest/build.py's wrapper exit codes.

## Expected behavior

`onnxruntime_provider_test` should complete normally. If a native failure occurs, CI should retain enough information (provider, NT status, and dump) to identify the crashing module and stack.

## System information

- Platform: Windows, self-hosted GitHub Actions runner with Visual Studio 2022
- Architecture: x64
- ONNX Runtime installation: built from source
- ONNX Runtime commit: `6c227c76265711189e15e08e5333430a29a897e7`
- Execution providers: CPU and OpenVINO
- OpenVINO version: 2026.2.1 (`ede283a88e3`)
- CPU: AMD EPYC 9V74 80-Core Processor
- Related run: https://github.com/microsoft/onnxruntime/actions/runs/31247477067

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/windows_openvino.yml and the MatMulNBits.Float32_4b_Accuracy0 entry point in onnxruntime_provider_test.exe. Run the isolated test repeatedly with --gtest_repeat=100, then use the suggested provider-specific variants, ONEDNN_VERBOSE=1, and Windows dump collection. Done means identifying the active execution provider, NT status, and crashing module or stack on recurrence.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, github-actions
Domain
ci-cd, machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.