intel / intel/auto-round

[Bug]: ARK SageV1 BF16 D128 causes DEVICE_LOST on Windows Arc B580

Open
#2,213 0 comments 0 reactions 1 assignee Claimed by @Zhenzhong1 View on GitHub
bug
Dominant language
Python
Stars
1.6k
Forks
175
Avg merge
1d 18h
Merged PRs (30d)
99

Description

### Problem Description

On Windows with an Intel Arc B580, the smallest BF16 SageV1 call I tested causes the Level Zero device to be lost at the first synchronization. The equivalent PyTorch SDPA call with the same Q/K/V tensors succeeds and returns finite values.

This does not appear to be an OOM: the reproduction uses only [1, 8, 128, 128] BF16 tensors, and a fresh Python process can run a BF16 matrix multiplication successfully immediately afterward.

I reproduced the same failure with the default Level Zero command-list behavior and with SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0.

### Reproduction Steps

Build ARK for XPU/SYCL-TLA and run:

~~~python
import math
import torch
import torch.nn.functional as F
import auto_round_kernel as ark

torch.manual_seed(20260819)
q = torch.randn((1, 8, 128, 128), device="xpu", dtype=torch.bfloat16)
k = torch.randn_like(q)
v = torch.randn_like(q)
torch.xpu.synchronize()

# Control passes and is finite.
ref = F.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=False)
torch.xpu.synchronize()
assert torch.isfinite(ref).all()

# The first synchronization after this call loses the device.
out = ark.sagev1(
q, k, v,
dropout_p=0.0,
is_causal=False,
scale=1.0 / math.sqrt(128),
quant_block_size=64,
tensor_layout="HND",
smooth_k=True,
)
torch.xpu.synchronize()
~~~

### Environment Information

- OS: Windows x64, build 26100
- GPU: Intel Arc B580 Graphics, 12 GiB
- GPU driver: 32.0.101.8974 (2026-08-11)
- Python: 3.13.12
- PyTorch: 2.13.0+xpu
- Torch XPU runtime: 20260000
- oneAPI DPC++ compiler: 2026.1.0, build 20260617
- Tested ARK minimal binary source commit: 674377982561707cdc51db99cd8ff634369e20b7
- SYCL-TLA: fedbba404c6fb634554b7e5aec9a463710b6f952 (260630)
- Current AutoRound main checked: b9f3d0079d014c73a1ff009800c597b9bc3f2a36

Current main still pins SYCL-TLA 260630. I also reviewed the newer ARK compiler-memory refactor ff2fd25c644d4c4ea385f2eaeb691fa6eb6edc29; it splits normal SDPA template instantiations but does not change the SageV1 D128 execution path used by this reproduction.

### Error Logs

```shell
RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
```

### Additional Context

- ARK import succeeds and exposes sagev1, sage, and sage_dynamic_quant.
- PyTorch SDPA with the exact same input shape and dtype succeeds.
- The failure also occurs when immediate command lists are disabled.
- A fresh process after the failure successfully completes a BF16 64x64 XPU matmul, so the GPU/driver recovers and there is no persistent device damage.
- The minimal build isolates BF16 SageV1, head dimension 128, and excludes unrelated kernels to fit compilation within 32 GiB system RAM. The executed SageV1/SYCL-TLA path is the same one still present in current main.

Is Windows + Battlemage G21 + BF16 SageV1 D128 expected to be supported with this version combination? If so, could you advise the supported driver/Torch/oneAPI matrix or point to a kernel fix/guard for this device-lost path?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.