intel / intel/llm-scaler

Dual Intel Arc B70: oneAPI 2025.3 L0 V2 multi-device context fails and blocks PyTorch/vLLM TP=2

Open
#463 4 comments 0 reactions 1 assignee Claimed by @james-tang17 View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

### Problem

The multi-BMG/offline installer version:

26.18.8.2
the GPU user-mode stack reports:

Level Zero / UR: 1.14.36300+8
OpenCL NEO: 25.48.36300.8
And host kernel:
6.17.0-1009-intel

On a dual Intel Arc Pro B70 / `Intel(R) Graphics [0xe223]` system, the oneAPI 2025.3 runtime stack fails when both GPUs are visible. This blocks PyTorch XPU multi-GPU enumeration, basic allocation, XCCL all-reduce, and ultimately vLLM tensor-parallel serving.

The workaround below is useful, but it is not the desired fix. The main issue is that the newer Level Zero V2 / Unified Runtime path appears broken for dual B-Series GPUs and should be investigated/fixed.

### Environment

- OS: Ubuntu 24.04
- Kernel: `6.17.0-1009-intel`
- GPUs: 2x `Intel(R) Graphics [0xe223]`
- vLLM: `0.22.1rc1.dev265+g94fcdd007.xpu`
- PyTorch/XPU versions tested:
- `torch 2.9.1+xpu` with `intel-sycl-rt 2025.2.1`: works
- `torch 2.10.0+xpu` with `intel-sycl-rt 2025.3.1`: fails
- `torch 2.11.0+xpu` with 2025.3.x stack: fails
- `torch 2.12.0+xpu` with `intel-sycl-rt 2025.3.2`: fails

### Version Matrix

```text
torch 2.9.1+xpu intel-sycl-rt 2025.2.1 PASS: torch.xpu.device_count() == 2
torch 2.10.0+xpu intel-sycl-rt 2025.3.1 FAIL: UR_RESULT_ERROR_UNKNOWN
torch 2.11.0+xpu 2025.3.x stack FAIL: UR_RESULT_ERROR_UNKNOWN
torch 2.12.0+xpu intel-sycl-rt 2025.3.2 FAIL: UR_RESULT_ERROR_UNKNOWN
```
#### Failure 1: L0 V2 adapter cannot create a multi-device SYCL context

Minimal SYCL repro:

std::vector gpus;
for (const auto& d : p.get_devices()) {
if (d.is_gpu()) {
gpus.push_back(d);
}
}

sycl::context multi(gpus);
Result with both GPUs visible:

ONEAPI_DEVICE_SELECTOR=level_zero:0,1 /tmp/sycl_context_modes
GPU: Intel(R) Graphics [0xe223]
GPU: Intel(R) Graphics [0xe223]
gpu count: 2
before single-device context 0
single-device context ok 0
before single-device context 1
single-device context ok 1
before explicit multi-device context
SYCL exception: level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)
So single-device contexts work, but a multi-device context fails.

Failure 2: PyTorch XPU device enumeration fails with the 2025.3 runtime stack
With both GPUs visible:

ONEAPI_DEVICE_SELECTOR=level_zero:0,1 python - <<'PY'
import torch
print(torch.__version__)
print(torch.xpu.is_available())
print(torch.xpu.device_count())
PY
Fails on torch 2.10.0+xpu, 2.11.0+xpu, and 2.12.0+xpu:

RuntimeError: level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)
This appears to correspond to the same multi-device context creation failure.

Failure 3: After disabling L0 V2, allocation still fails unless USM residency is disabled
With:

SYCL_UR_USE_LEVEL_ZERO_V2=0
ONEAPI_DEVICE_SELECTOR=level_zero:0,1
PyTorch can report two devices, but a tiny allocation fails:

x = torch.zeros(1, device="xpu")
Error:

torch.OutOfMemoryError: XPU out of memory. Tried to allocate 2.00 MiB.
GPU has 30.30 GiB free.
Adding this avoids the allocation failure:

SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0
Failure 4: oneCCL/XCCL all-reduce fails with Level Zero OOM
Even after fixing context creation and allocation with:

SYCL_UR_USE_LEVEL_ZERO_V2=0
SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0
ONEAPI_DEVICE_SELECTOR=level_zero:0,1
A minimal XCCL all-reduce fails:

import os
import torch
import torch.distributed as dist

rank = int(os.environ["LOCAL_RANK"])
torch.xpu.set_device(rank)
dist.init_process_group(backend="xccl")

x = torch.ones(1, device="xpu")
dist.all_reduce(x)
torch.xpu.synchronize()
print("after", rank, x, flush=True)
Error:

RuntimeError: level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)
This only works if oneCCL SYCL kernels are disabled and direct all-reduce is forced:

CCL_ENABLE_SYCL_KERNELS=0
CCL_ALLREDUCE=direct
Workaround
The following workaround allows basic PyTorch XPU allocation and XCCL all-reduce to pass:

export SYCL_UR_USE_LEVEL_ZERO_V2=0
export SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0
export CCL_ENABLE_SYCL_KERNELS=0
export CCL_ALLREDUCE=direct
export ONEAPI_DEVICE_SELECTOR=level_zero:0,1
With those variables:

torch.xpu.device_count() -> 2
torch.zeros(1, device="xpu:0") -> OK
torch.zeros(1, device="xpu:1") -> OK
xccl all_reduce tensor([1.]) across two ranks -> tensor([2.])
Why this is not sufficient
The workaround disables the L0 V2 adapter and disables oneCCL SYCL-kernel collectives, so it is not an acceptable long-term solution. It likely loses performance and disables the newer code paths that should work on B-Series GPUs.

Request
Please investigate the underlying issues in the oneAPI 2025.3 stack for dual B-Series GPUs:

L0 V2 adapter fails multi-device SYCL context creation.
Legacy L0 adapter requires SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0 to avoid false OOM on tiny allocations.
oneCCL SYCL-kernel all-reduce fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY.
These issues block vLLM tensor-parallel serving on dual Intel Arc B70 unless the workaround is applied.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.