intel / intel/compute-runtime

[GSD-12973] Arc Pro B70 4-GPU: SYCL/Level Zero peer memcpy corrupts data and causes DEVICE_LOST

Open
#942 7 comments 0 reactions 0 assignees View on GitHub
OS: Linux Type: Bug
Dominant language
C++
Stars
1.4k
Forks
300
PR merge metrics
No merged PRs in 30d

Description

# Arc Pro B70 4-GPU: SYCL/Level Zero peer memcpy corrupts data and causes DEVICE_LOST

## Summary

On a 4x Intel Arc Pro B70 system, direct SYCL/Level Zero device-to-device copies between GPUs corrupt data. The runtime reports peer access is available via `ext_oneapi_can_access_peer(...)`, but actual `queue.memcpy(dst_device_ptr, src_device_ptr, size)` returns incorrect bytes. Larger copies can trigger `UR_RESULT_ERROR_DEVICE_LOST`.

Host-staged copies (`GPU -> host RAM -> GPU`) work correctly.

This breaks llama.cpp multi-GPU inference unless cross-device copies are forced through host staging. With direct dGPU-to-dGPU copies enabled, model output becomes gibberish.

## Hardware / Topology

- 4x Intel Arc Pro B70 GPUs
- Ubuntu 26.04 LTS
- Kernel: `7.0.0-27-generic`
- Driver: `xe`
- PCIe fabric / switch topology with four B70s attached under the same root complex

PCIe GPU mapping:

```text
0f:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD128
13:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD129
19:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD130
1d:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD131
```

Relevant topology excerpt:

```text
03.1-[09-1f]----00.0-[0a-1f]
+-00.0-[0b-14]
| +-10.0-[0d-10] -> 0f:00.0 Intel Battlemage G31
| +-18.0-[11-14] -> 13:00.0 Intel Battlemage G31
+-04.0-[15-1e]
| +-00.0-[17-1a] -> 19:00.0 Intel Battlemage G31
| +-08.0-[1b-1e] -> 1d:00.0 Intel Battlemage G31
```

Full `lspci -tv` output is available in `lspci-tv-b70-bug.txt`.

## Software Versions

```text
intel-igc-core-2: 2.36.3
intel-igc-opencl-2: 2.36.3
intel-ocloc: 26.22.38646.4-0
intel-opencl-icd: 26.22.38646.4-0
libigdgmm12: 22.10.0
libze-intel-gpu1: 26.22.38646.4-0
Level Zero driver: 1.15.38646+4
```

`sycl-ls` sees all four GPUs:

```text
[level_zero:gpu][level_zero:0] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
[level_zero:gpu][level_zero:1] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
[level_zero:gpu][level_zero:2] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
[level_zero:gpu][level_zero:3] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
```

## Reproduction

I used a minimal SYCL repro that:

1. Enumerates Level Zero GPU devices.
2. Allocates a source USM buffer on GPU A.
3. Allocates a destination USM buffer on GPU B.
4. Fills the source with deterministic bytes.
5. Runs direct `queue.memcpy(dst, src, size)` on the destination queue.
6. Copies destination back to host and compares bytes.

Run command:

```bash
source /opt/intel/oneapi/setvars.sh
ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
ZES_ENABLE_SYSMAN=1 \
UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 \
./sycl_peer_copy_probe
```

Failure example:

```text
level_zero_gpu_count=4
gpu[0]=Intel(R) Arc(TM) Pro B70 Graphics
gpu[1]=Intel(R) Arc(TM) Pro B70 Graphics
gpu[2]=Intel(R) Arc(TM) Pro B70 Graphics
gpu[3]=Intel(R) Arc(TM) Pro B70 Graphics
pair 0->1 can_access=1
FAIL 0->1 size=1 first_bad=0 expected=0xaf actual=0xff
FAIL 0->1 size=2 first_bad=0 expected=0xc1 actual=0xff
FAIL 0->1 size=3 first_bad=0 expected=0xce actual=0xff
FAIL 0->1 size=4 first_bad=0 expected=0x62 actual=0xff
FAIL 0->1 size=7 first_bad=0 expected=0x21 actual=0xff
FAIL 0->1 size=16 first_bad=0 expected=0x9f actual=0xff
FAIL 0->1 size=31 first_bad=0 expected=0x13 actual=0xff
FAIL 0->1 size=64 first_bad=0 expected=0x35 actual=0xff
FAIL 0->1 size=255 first_bad=0 expected=0x29 actual=0xff
FAIL 0->1 size=4096 first_bad=0 expected=0x8d actual=0xff
FAIL 0->1 size=65536 first_bad=0 expected=0x76 actual=0xff
FAIL 0->1 size=1048576 first_bad=0 expected=0x6c actual=0xff
FAIL 0->1 size=16777216 exception=level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
FAIL 0->1 size=16777216 first_bad=0 expected=0xa6 actual=0x00
Abort was called at 284 line in file:
../../neo/shared/source/os_interface/linux/drm_neo.cpp
```

After the `DEVICE_LOST` failure, `sycl-ls` may temporarily show only 3 of the 4 Level Zero GPUs until reboot.

## Expected Behavior

If `ext_oneapi_can_access_peer(...)` returns true, direct peer copies between those GPU devices should either:

- copy bytes correctly, or
- fail cleanly before corrupting data.

## Actual Behavior

Direct peer copies return corrupted bytes. Larger copies can cause `UR_RESULT_ERROR_DEVICE_LOST`.

## Workaround

Host-staged copies work:

```text
GPU A -> host RAM -> GPU B
```

In llama.cpp, forcing host-staged cross-device copies avoids gibberish output:

```bash
GGML_SYCL_DEV2DEV_MEMCPY=2
```

## Impact

This prevents reliable multi-GPU inference on 4x Arc Pro B70 when using direct dGPU-to-dGPU copies. It removes the expected benefit of the PCIe fabric for peer transfers and requires slower host-staged copies for correctness.

## Attachments / Extra Outputs

I can provide:

- `sycl_peer_copy_probe.cpp`
- `sycl_peer_copy_shared_context_probe.cpp`
- `sycl-ls-b70-bug.txt`
- `llama-ls-sycl-device-b70-bug.txt`
- `lspci-tv-b70-bug.txt`
- `packages-b70-bug.txt`
- `uname-b70-bug.txt`

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied sycl_peer_copy_probe.cpp reproduction with the listed ONEAPI_DEVICE_SELECTOR and environment variables, then compare direct peer copies with the documented host-staged workaround. Use sycl_peer_copy_shared_context_probe.cpp and the attached topology and package outputs to narrow the failing configuration. Done means peer copies either transfer bytes correctly or fail cleanly without DEVICE_LOST or GPU disappearance.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, computer-graphics
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.