Windows: full pytest suite crashes deterministically at ~95% — access violation in CPU-side torch reference (order-dependent native memory corruption)
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
### Summary
On Windows (Arc Pro B60, ComfyUI portable env), the **full `omni_xpu_kernel` test suite cannot complete in a single `pytest` process**: a deterministic access violation lands in a **pure-CPU PyTorch op** — `torch.where` on int16 at `tests/test_svdq_correctness.py:34` (`ref_unpack_int4`) — after ~684 of 720 collected tests (~95%). The faulting function is a CPU reference helper with no kernel involvement, and the crash only appears after full-suite accumulation, which points at native memory corruption by earlier native code surfacing in an innocent victim.
Source: omni 0.2.0-b1 tree at `b9b0c4c900f1`, kernel built from source per `omni/docs/WINDOWS_PORTABLE.md` (DPC++ 2025.3.1, wheel `omni_xpu_kernel-0.1.0b9.dev0+torch212.bmg`).
### Steps to reproduce
(kernel installed into the Python env; clean shell, no oneAPI env vars set)
```
cd omni/omni_xpu_kernel/tests
python -X faulthandler -m pytest -q .
```
### Actual
Crash at test 685/720 (~95%), identical position across consecutive runs:
```
Windows fatal exception: access violation
Current thread (most recent call first):
File "...tests\test_svdq_correctness.py", line 34 in ref_unpack_int4
File "...tests\test_svdq_correctness.py", line 50 in ref_dequantize_w4
File "...tests\test_svdq_correctness.py", line 124 in test_dequantize_matches_reference
```
Victim: `test_dequantize_matches_reference[11520-3840-float32]` (largest nunchaku QKV shape). The faulting code is plain CPU torch:
```python
def ref_unpack_int4(packed, signed=True):
p = packed.view(torch.uint8).to(torch.int16)
low = p & 0x0F
high = (p >> 4) & 0x0F
low = torch.where(low >= 8, low - 16, low) # <-- line 34, AV here
...
```
### Evidence the crash is order-dependent (not test-local)
- `test_svdq_correctness.py` alone: **25/25 pass** (3.4 s).
- **All 21 pairwise combinations** of the svdq file with each of the other 20 test files: pass.
- Full suite: deterministic crash — same test, same ~95% position, across runs.
- Deselecting the victim test: the process **still crashes at ~95%**, in a *different* svdq reference helper at the same position — the crash floats to whichever CPU-side svdq reference code runs at that point in the sequence.
No single-file or pairwise trigger exists; the fault surfaces in plain CPU torch ops only after ~640+ tests of accumulated native/XPU activity. Real-world ComfyUI exposure is likely lower (diffusion workloads hammer a small set of op shapes rather than 720 tests of varied native paths), but the suite itself cannot complete on Windows here.
### Other observation (minor, likely unrelated)
```
FAILED test_kitchen_rope.py::test_h3_packed_qkv_partial_rms_rope_inplace
Mismatched elements: 1 / 265216 (0.0%)
Greatest absolute difference: 0.0234375 at index (0, 26, 12, 76) (up to 0.02 allowed)
```
Single-element tolerance miss, deterministic across three runs; 0.0234375 = 3/128, which reads like borderline bf16 rounding on this part rather than anything related to the crash.
### Environment
| Component | Version |
|---|---|
| OS | Windows 11 23H2 (10.0.22631) |
| GPU | Intel Arc Pro B60, driver 32.0.101.8805 |
| Python | 3.13.12 (ComfyUI portable `python_embeded`) |
| torch | 2.12.0+xpu |
| Build | `omni/omni_xpu_kernel` @ b9b0c4c900f1, DPC++ 2025.3.1 |
| pytest | 9.1.1 |
Related: #611 (independent Windows report on B580, torch-2.13/oneAPI-2026.1 generation). A second Windows finding from the same validation run — lazy `dnnl.dll` resolution picking the wrong oneAPI install — is filed separately.
Full logs and faulthandler dumps available on request.
Contributor guide
Assessment
This issue has not been assessed yet.