intel / intel/llm-scaler

Windows: lazy dnnl.dll resolution picks newer oneAPI install on multi-install systems — every oneDNN engine creation fails ('could not create an engine')

Open
#625 1 comment 0 reactions 1 assignee Claimed by @xiangyuT View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

### Summary

On Windows systems with a newer oneAPI (e.g. 2026.0) installed alongside the documented 2025.3 build toolchain, `omni_xpu_kernel._C`'s lazy `LoadLibrary("dnnl.dll")` at first oneDNN engine creation resolves to the **newer** oneAPI's dnnl (3.11, sycl9-era), which cannot interop with the sycl8-generation runtime used by the kernel and by torch 2.12.0+xpu. Result: **every oneDNN-dependent test fails** with `RuntimeError: could not create an engine` (~113 failures here), while non-oneDNN kernels work fine.

**PATH ordering does not fix this** — including prepending the build-matched bin dir from inside Python before the first oneDNN call.

### Symptom

```
omni_xpu_kernel/linear/__init__.py:83: RuntimeError: could not create an engine
```

oneDNN verbose (`DNNL_VERBOSE=1`):

```
dnnl_verbose,error,runtime,bad engine kind,src\xpu\sycl\capi\capi_engine.cpp:45
after call: dnnl = C:\Program Files (x86)\Intel\oneAPI\dnnl\2026.0\bin\dnnl.dll
```

### Cause

The kernel, built per `omni/docs/WINDOWS_PORTABLE.md` (DPC++ 2025.3.1), links the sycl8-generation runtime; torch 2.12.0+xpu's `torch_xpu.dll`/`c10_xpu.dll` likewise link sycl8. With oneAPI 2026.0 also present on the machine, the OS loader resolves the by-name `LoadLibrary("dnnl.dll")` to 2026.0's dnnl 3.11, whose engine rejects the older sycl queue ("bad engine kind").

### Workaround that works

Pre-load the build-matched dnnl by full path before the first oneDNN call — Windows then reuses the already-mapped module for `_C`'s later by-name load:

```python
import ctypes
ctypes.CDLL(r"\Library\bin\dnnl.dll") # build-matched dnnl (3.9.1 here)
import omni_xpu_kernel # engine creation now succeeds
```

With this pre-load, the fp8/oneDNN blocks go green and an FP8 GEMM (W8A16) runs numerically correct on Arc Pro B60 (max |diff| 0.0078 vs fp32 reference).

### Suggestion

Ship the pre-load (or `LoadLibraryExW` with the fully resolved path and `LOAD_WITH_ALTERED_SEARCH_PATH`) in the package's Windows loader path, preferring the dnnl that sits beside the installed torch/runtime (`python_root\Library\bin`) over any system oneAPI. Alternatively, resolve dnnl relative to torch's own vendored runtime.

### Environment

| Component | Version |
|---|---|
| OS | Windows 11 23H2 (10.0.22631) |
| GPU | Intel Arc Pro B60, driver 32.0.101.8805 |
| Python | 3.13.12 (ComfyUI portable `python_embeded`) |
| torch | 2.12.0+xpu (sycl8-generation runtime) |
| oneAPI installs | 2026.0 (system) + 2025.3 (build toolchain), side-by-side |
| Build | `omni/omni_xpu_kernel` @ b9b0c4c900f1, DPC++ 2025.3.1 |

Related: #611 (B580 report on the torch-2.13/oneAPI-2026.1 generation — the same DLL-generation mismatch family approached from the other toolchain direction). The full-suite order-dependent crash found in the same validation run is filed separately.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.