Dual Intel Arc B70: oneAPI 2025.3 L0 V2 multi-device context fails and blocks PyTorch/vLLM TP=2
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
### Problem
The multi-BMG/offline installer version:
26.18.8.2
the GPU user-mode stack reports:
Level Zero / UR: 1.14.36300+8
OpenCL NEO: 25.48.36300.8
And host kernel:
6.17.0-1009-intel
On a dual Intel Arc Pro B70 / `Intel(R) Graphics [0xe223]` system, the oneAPI 2025.3 runtime stack fails when both GPUs are visible. This blocks PyTorch XPU multi-GPU enumeration, basic allocation, XCCL all-reduce, and ultimately vLLM tensor-parallel serving.
The workaround below is useful, but it is not the desired fix. The main issue is that the newer Level Zero V2 / Unified Runtime path appears broken for dual B-Series GPUs and should be investigated/fixed.
### Environment
- OS: Ubuntu 24.04
- Kernel: `6.17.0-1009-intel`
- GPUs: 2x `Intel(R) Graphics [0xe223]`
- vLLM: `0.22.1rc1.dev265+g94fcdd007.xpu`
- PyTorch/XPU versions tested:
- `torch 2.9.1+xpu` with `intel-sycl-rt 2025.2.1`: works
- `torch 2.10.0+xpu` with `intel-sycl-rt 2025.3.1`: fails
- `torch 2.11.0+xpu` with 2025.3.x stack: fails
- `torch 2.12.0+xpu` with `intel-sycl-rt 2025.3.2`: fails
### Version Matrix
```text
torch 2.9.1+xpu intel-sycl-rt 2025.2.1 PASS: torch.xpu.device_count() == 2
torch 2.10.0+xpu intel-sycl-rt 2025.3.1 FAIL: UR_RESULT_ERROR_UNKNOWN
torch 2.11.0+xpu 2025.3.x stack FAIL: UR_RESULT_ERROR_UNKNOWN
torch 2.12.0+xpu intel-sycl-rt 2025.3.2 FAIL: UR_RESULT_ERROR_UNKNOWN
```
#### Failure 1: L0 V2 adapter cannot create a multi-device SYCL context
Minimal SYCL repro:
std::vector gpus;
for (const auto& d : p.get_devices()) {
if (d.is_gpu()) {
gpus.push_back(d);
}
}
sycl::context multi(gpus);
Result with both GPUs visible:
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 /tmp/sycl_context_modes
GPU: Intel(R) Graphics [0xe223]
GPU: Intel(R) Graphics [0xe223]
gpu count: 2
before single-device context 0
single-device context ok 0
before single-device context 1
single-device context ok 1
before explicit multi-device context
SYCL exception: level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)
So single-device contexts work, but a multi-device context fails.
Failure 2: PyTorch XPU device enumeration fails with the 2025.3 runtime stack
With both GPUs visible:
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 python - <<'PY'
import torch
print(torch.__version__)
print(torch.xpu.is_available())
print(torch.xpu.device_count())
PY
Fails on torch 2.10.0+xpu, 2.11.0+xpu, and 2.12.0+xpu:
RuntimeError: level_zero backend failed with error: 2147483646 (UR_RESULT_ERROR_UNKNOWN)
This appears to correspond to the same multi-device context creation failure.
Failure 3: After disabling L0 V2, allocation still fails unless USM residency is disabled
With:
SYCL_UR_USE_LEVEL_ZERO_V2=0
ONEAPI_DEVICE_SELECTOR=level_zero:0,1
PyTorch can report two devices, but a tiny allocation fails:
x = torch.zeros(1, device="xpu")
Error:
torch.OutOfMemoryError: XPU out of memory. Tried to allocate 2.00 MiB.
GPU has 30.30 GiB free.
Adding this avoids the allocation failure:
SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0
Failure 4: oneCCL/XCCL all-reduce fails with Level Zero OOM
Even after fixing context creation and allocation with:
SYCL_UR_USE_LEVEL_ZERO_V2=0
SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0
ONEAPI_DEVICE_SELECTOR=level_zero:0,1
A minimal XCCL all-reduce fails:
import os
import torch
import torch.distributed as dist
rank = int(os.environ["LOCAL_RANK"])
torch.xpu.set_device(rank)
dist.init_process_group(backend="xccl")
x = torch.ones(1, device="xpu")
dist.all_reduce(x)
torch.xpu.synchronize()
print("after", rank, x, flush=True)
Error:
RuntimeError: level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)
This only works if oneCCL SYCL kernels are disabled and direct all-reduce is forced:
CCL_ENABLE_SYCL_KERNELS=0
CCL_ALLREDUCE=direct
Workaround
The following workaround allows basic PyTorch XPU allocation and XCCL all-reduce to pass:
export SYCL_UR_USE_LEVEL_ZERO_V2=0
export SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0
export CCL_ENABLE_SYCL_KERNELS=0
export CCL_ALLREDUCE=direct
export ONEAPI_DEVICE_SELECTOR=level_zero:0,1
With those variables:
torch.xpu.device_count() -> 2
torch.zeros(1, device="xpu:0") -> OK
torch.zeros(1, device="xpu:1") -> OK
xccl all_reduce tensor([1.]) across two ranks -> tensor([2.])
Why this is not sufficient
The workaround disables the L0 V2 adapter and disables oneCCL SYCL-kernel collectives, so it is not an acceptable long-term solution. It likely loses performance and disables the newer code paths that should work on B-Series GPUs.
Request
Please investigate the underlying issues in the oneAPI 2025.3 stack for dual B-Series GPUs:
L0 V2 adapter fails multi-device SYCL context creation.
Legacy L0 adapter requires SYCL_PI_LEVEL_ZERO_USM_RESIDENT=0 to avoid false OOM on tiny allocations.
oneCCL SYCL-kernel all-reduce fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY.
These issues block vLLM tensor-parallel serving on dual Intel Arc B70 unless the workaround is applied.
Contributor guide
Assessment
This issue has not been assessed yet.