intel / intel/llm-scaler

Windows: full pytest suite crashes deterministically at ~95% — access violation in CPU-side torch reference (order-dependent native memory corruption)

Open
#624 1 comment 0 reactions 1 assignee Claimed by @xiangyuT View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

### Summary

On Windows (Arc Pro B60, ComfyUI portable env), the **full `omni_xpu_kernel` test suite cannot complete in a single `pytest` process**: a deterministic access violation lands in a **pure-CPU PyTorch op** — `torch.where` on int16 at `tests/test_svdq_correctness.py:34` (`ref_unpack_int4`) — after ~684 of 720 collected tests (~95%). The faulting function is a CPU reference helper with no kernel involvement, and the crash only appears after full-suite accumulation, which points at native memory corruption by earlier native code surfacing in an innocent victim.

Source: omni 0.2.0-b1 tree at `b9b0c4c900f1`, kernel built from source per `omni/docs/WINDOWS_PORTABLE.md` (DPC++ 2025.3.1, wheel `omni_xpu_kernel-0.1.0b9.dev0+torch212.bmg`).

### Steps to reproduce

(kernel installed into the Python env; clean shell, no oneAPI env vars set)

```
cd omni/omni_xpu_kernel/tests
python -X faulthandler -m pytest -q .
```

### Actual

Crash at test 685/720 (~95%), identical position across consecutive runs:

```
Windows fatal exception: access violation
Current thread (most recent call first):
File "...tests\test_svdq_correctness.py", line 34 in ref_unpack_int4
File "...tests\test_svdq_correctness.py", line 50 in ref_dequantize_w4
File "...tests\test_svdq_correctness.py", line 124 in test_dequantize_matches_reference
```

Victim: `test_dequantize_matches_reference[11520-3840-float32]` (largest nunchaku QKV shape). The faulting code is plain CPU torch:

```python
def ref_unpack_int4(packed, signed=True):
p = packed.view(torch.uint8).to(torch.int16)
low = p & 0x0F
high = (p >> 4) & 0x0F
low = torch.where(low >= 8, low - 16, low) # <-- line 34, AV here
...
```

### Evidence the crash is order-dependent (not test-local)

- `test_svdq_correctness.py` alone: **25/25 pass** (3.4 s).
- **All 21 pairwise combinations** of the svdq file with each of the other 20 test files: pass.
- Full suite: deterministic crash — same test, same ~95% position, across runs.
- Deselecting the victim test: the process **still crashes at ~95%**, in a *different* svdq reference helper at the same position — the crash floats to whichever CPU-side svdq reference code runs at that point in the sequence.

No single-file or pairwise trigger exists; the fault surfaces in plain CPU torch ops only after ~640+ tests of accumulated native/XPU activity. Real-world ComfyUI exposure is likely lower (diffusion workloads hammer a small set of op shapes rather than 720 tests of varied native paths), but the suite itself cannot complete on Windows here.

### Other observation (minor, likely unrelated)

```
FAILED test_kitchen_rope.py::test_h3_packed_qkv_partial_rms_rope_inplace
Mismatched elements: 1 / 265216 (0.0%)
Greatest absolute difference: 0.0234375 at index (0, 26, 12, 76) (up to 0.02 allowed)
```

Single-element tolerance miss, deterministic across three runs; 0.0234375 = 3/128, which reads like borderline bf16 rounding on this part rather than anything related to the crash.

### Environment

| Component | Version |
|---|---|
| OS | Windows 11 23H2 (10.0.22631) |
| GPU | Intel Arc Pro B60, driver 32.0.101.8805 |
| Python | 3.13.12 (ComfyUI portable `python_embeded`) |
| torch | 2.12.0+xpu |
| Build | `omni/omni_xpu_kernel` @ b9b0c4c900f1, DPC++ 2025.3.1 |
| pytest | 9.1.1 |

Related: #611 (independent Windows report on B580, torch-2.13/oneAPI-2026.1 generation). A second Windows finding from the same validation run — lazy `dnnl.dll` resolution picking the wrong oneAPI install — is filed separately.

Full logs and faulthandler dumps available on request.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.