[GSD-12973] Arc Pro B70 4-GPU: SYCL/Level Zero peer memcpy corrupts data and causes DEVICE_LOST
- Dominant language
- C++
- Stars
- 1.4k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
# Arc Pro B70 4-GPU: SYCL/Level Zero peer memcpy corrupts data and causes DEVICE_LOST
## Summary
On a 4x Intel Arc Pro B70 system, direct SYCL/Level Zero device-to-device copies between GPUs corrupt data. The runtime reports peer access is available via `ext_oneapi_can_access_peer(...)`, but actual `queue.memcpy(dst_device_ptr, src_device_ptr, size)` returns incorrect bytes. Larger copies can trigger `UR_RESULT_ERROR_DEVICE_LOST`.
Host-staged copies (`GPU -> host RAM -> GPU`) work correctly.
This breaks llama.cpp multi-GPU inference unless cross-device copies are forced through host staging. With direct dGPU-to-dGPU copies enabled, model output becomes gibberish.
## Hardware / Topology
- 4x Intel Arc Pro B70 GPUs
- Ubuntu 26.04 LTS
- Kernel: `7.0.0-27-generic`
- Driver: `xe`
- PCIe fabric / switch topology with four B70s attached under the same root complex
PCIe GPU mapping:
```text
0f:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD128
13:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD129
19:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD130
1d:00.0 Intel Battlemage G31 / Arc Pro B70 -> renderD131
```
Relevant topology excerpt:
```text
03.1-[09-1f]----00.0-[0a-1f]
+-00.0-[0b-14]
| +-10.0-[0d-10] -> 0f:00.0 Intel Battlemage G31
| +-18.0-[11-14] -> 13:00.0 Intel Battlemage G31
+-04.0-[15-1e]
| +-00.0-[17-1a] -> 19:00.0 Intel Battlemage G31
| +-08.0-[1b-1e] -> 1d:00.0 Intel Battlemage G31
```
Full `lspci -tv` output is available in `lspci-tv-b70-bug.txt`.
## Software Versions
```text
intel-igc-core-2: 2.36.3
intel-igc-opencl-2: 2.36.3
intel-ocloc: 26.22.38646.4-0
intel-opencl-icd: 26.22.38646.4-0
libigdgmm12: 22.10.0
libze-intel-gpu1: 26.22.38646.4-0
Level Zero driver: 1.15.38646+4
```
`sycl-ls` sees all four GPUs:
```text
[level_zero:gpu][level_zero:0] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
[level_zero:gpu][level_zero:1] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
[level_zero:gpu][level_zero:2] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
[level_zero:gpu][level_zero:3] Intel(R) Arc(TM) Pro B70 Graphics 20.2.0 [1.15.38646+4]
```
## Reproduction
I used a minimal SYCL repro that:
1. Enumerates Level Zero GPU devices.
2. Allocates a source USM buffer on GPU A.
3. Allocates a destination USM buffer on GPU B.
4. Fills the source with deterministic bytes.
5. Runs direct `queue.memcpy(dst, src, size)` on the destination queue.
6. Copies destination back to host and compares bytes.
Run command:
```bash
source /opt/intel/oneapi/setvars.sh
ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
ZES_ENABLE_SYSMAN=1 \
UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 \
./sycl_peer_copy_probe
```
Failure example:
```text
level_zero_gpu_count=4
gpu[0]=Intel(R) Arc(TM) Pro B70 Graphics
gpu[1]=Intel(R) Arc(TM) Pro B70 Graphics
gpu[2]=Intel(R) Arc(TM) Pro B70 Graphics
gpu[3]=Intel(R) Arc(TM) Pro B70 Graphics
pair 0->1 can_access=1
FAIL 0->1 size=1 first_bad=0 expected=0xaf actual=0xff
FAIL 0->1 size=2 first_bad=0 expected=0xc1 actual=0xff
FAIL 0->1 size=3 first_bad=0 expected=0xce actual=0xff
FAIL 0->1 size=4 first_bad=0 expected=0x62 actual=0xff
FAIL 0->1 size=7 first_bad=0 expected=0x21 actual=0xff
FAIL 0->1 size=16 first_bad=0 expected=0x9f actual=0xff
FAIL 0->1 size=31 first_bad=0 expected=0x13 actual=0xff
FAIL 0->1 size=64 first_bad=0 expected=0x35 actual=0xff
FAIL 0->1 size=255 first_bad=0 expected=0x29 actual=0xff
FAIL 0->1 size=4096 first_bad=0 expected=0x8d actual=0xff
FAIL 0->1 size=65536 first_bad=0 expected=0x76 actual=0xff
FAIL 0->1 size=1048576 first_bad=0 expected=0x6c actual=0xff
FAIL 0->1 size=16777216 exception=level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
FAIL 0->1 size=16777216 first_bad=0 expected=0xa6 actual=0x00
Abort was called at 284 line in file:
../../neo/shared/source/os_interface/linux/drm_neo.cpp
```
After the `DEVICE_LOST` failure, `sycl-ls` may temporarily show only 3 of the 4 Level Zero GPUs until reboot.
## Expected Behavior
If `ext_oneapi_can_access_peer(...)` returns true, direct peer copies between those GPU devices should either:
- copy bytes correctly, or
- fail cleanly before corrupting data.
## Actual Behavior
Direct peer copies return corrupted bytes. Larger copies can cause `UR_RESULT_ERROR_DEVICE_LOST`.
## Workaround
Host-staged copies work:
```text
GPU A -> host RAM -> GPU B
```
In llama.cpp, forcing host-staged cross-device copies avoids gibberish output:
```bash
GGML_SYCL_DEV2DEV_MEMCPY=2
```
## Impact
This prevents reliable multi-GPU inference on 4x Arc Pro B70 when using direct dGPU-to-dGPU copies. It removes the expected benefit of the PCIe fabric for peer transfers and requires slower host-staged copies for correctness.
## Attachments / Extra Outputs
I can provide:
- `sycl_peer_copy_probe.cpp`
- `sycl_peer_copy_shared_context_probe.cpp`
- `sycl-ls-b70-bug.txt`
- `llama-ls-sycl-device-b70-bug.txt`
- `lspci-tv-b70-bug.txt`
- `packages-b70-bug.txt`
- `uname-b70-bug.txt`
Contributor guide
Research direction
Start by running the supplied sycl_peer_copy_probe.cpp reproduction with the listed ONEAPI_DEVICE_SELECTOR and environment variables, then compare direct peer copies with the documented host-staged workaround. Use sycl_peer_copy_shared_context_probe.cpp and the attached topology and package outputs to narrow the failing configuration. Done means peer copies either transfer bytes correctly or fail cleanly without DEVICE_LOST or GPU disappearance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, computer-graphics
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100