microsoft / microsoft/onnxruntime
[OpenVINO][Windows] onnxruntime_provider_test silently exits during MatMulNBits.Float32_4b_Accuracy0
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
## Describe the issue
The Windows OpenVINO CI job [`93078339501`](https://github.com/microsoft/onnxruntime/actions/runs/31247477067/job/93078339501) failed because `onnxruntime_provider_test.exe` terminated silently while running `MatMulNBits.Float32_4b_Accuracy0`.
This was not a logged numerical accuracy failure:
- `MatMulNBits.Float32_4b_Accuracy0` started at `2026-08-08T08:21:30.4228366Z`.
- About 4.1 seconds later, CTest reported `onnxruntime_provider_test` as failed.
- There was no `[ OK ]` or `[ FAILED ]` result for the test, no gtest assertion, no C++ exception, and no Windows exception/status code in the log.
- CTest reported `91% tests passed, 1 tests failed out of 11`, returned exit status 8, and `build.py` consequently returned 1.
- The NPU availability skips and expected negative-test error logs earlier in the job were unrelated.
The retry, job [`93163934771`](https://github.com/microsoft/onnxruntime/actions/runs/31247477067/job/93163934771), passed the same test in 5.117 seconds and completed `onnxruntime_provider_test` successfully.
### Environment comparison
| | Failed attempt | Successful retry |
|---|---|---|
| ORT commit | `6c227c76265711189e15e08e5333430a29a897e7` | Same |
| OpenVINO | `2026.2.1` | Same |
| OpenVINO package commit | `ede283a88e3` | Same |
| CPU | AMD EPYC 9V74 80-Core Processor | Same |
| Runner | `4f589377c000005` | `862958acc000001` |
The observed failure is therefore intermittent and runner-instance-specific so far.
### Source analysis
- The workflow downloads `openvino_toolkit_windows_vc_mt_2026.2.1.21919.ede283a88e3_x86_64.zip`; commit `ede283a88e3` matches the OpenVINO `2026.2.1` tag.
- OpenVINO's `com.microsoft::MatMulNBits` frontend implementation is identical between tags `2026.1.0` and `2026.2.1`. It lowers the op to `u4 Convert -> Subtract -> Multiply -> Reshape/Slice -> MatMul`.
- The relevant OpenVINO CPU conversion/FullyConnected source is also unchanged between those versions. The downstream oneDNN revision does differ (`6b6492b1` to `87f65fdd`), including FP32 matmul dispatch changes, but there is not enough evidence to identify that as the cause.
- `OpTester` runs registered providers separately, CPU first and OpenVINO later. Because the process exited without an exception or provider message, the log does not identify which EP was active at termination.
- No crash dump or test-result artifact was uploaded by the failed run.
PR [#31759](https://github.com/microsoft/onnxruntime/pull/31759) proposes pinning Windows OpenVINO CI to 2026.1.0. That may be a temporary mitigation, but the current evidence does not establish a deterministic OpenVINO 2026.2.1 regression, and the MatMulNBits frontend itself is unchanged between the two versions.
## To reproduce
The failure is not yet reliably reproducible:
1. Build commit `6c227c76265711189e15e08e5333430a29a897e7` on Windows x64 with OpenVINO 2026.2.1 using `.github/workflows/windows_openvino.yml`.
2. Run `onnxruntime_provider_test.exe --gtest_filter=MatMulNBits.Float32_4b_Accuracy0` repeatedly.
3. The first CI attempt terminated during the test; the retry passed on another runner with the same CPU model.
Suggested diagnostics for the next recurrence:
1. Add temporary explicit CPU-only and OpenVINO-only variants to identify the active EP.
2. Run the isolated test with `--gtest_repeat=100`.
3. Enable Windows Error Reporting LocalDumps or ProcDump for `onnxruntime_provider_test.exe`, and upload full dumps as workflow artifacts.
4. Set `ONEDNN_VERBOSE=1` to capture the selected primitive and effective ISA before termination.
5. Record the child process NT exit status rather than only CTest/build.py's wrapper exit codes.
## Expected behavior
`onnxruntime_provider_test` should complete normally. If a native failure occurs, CI should retain enough information (provider, NT status, and dump) to identify the crashing module and stack.
## System information
- Platform: Windows, self-hosted GitHub Actions runner with Visual Studio 2022
- Architecture: x64
- ONNX Runtime installation: built from source
- ONNX Runtime commit: `6c227c76265711189e15e08e5333430a29a897e7`
- Execution providers: CPU and OpenVINO
- OpenVINO version: 2026.2.1 (`ede283a88e3`)
- CPU: AMD EPYC 9V74 80-Core Processor
- Related run: https://github.com/microsoft/onnxruntime/actions/runs/31247477067
Contributor guide
Research direction
Start with .github/workflows/windows_openvino.yml and the MatMulNBits.Float32_4b_Accuracy0 entry point in onnxruntime_provider_test.exe. Run the isolated test repeatedly with --gtest_repeat=100, then use the suggested provider-specific variants, ONEDNN_VERBOSE=1, and Windows dump collection. Done means identifying the active execution provider, NT status, and crashing module or stack on recurrence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, github-actions
- Domain
- ci-cd, machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100