intel / intel/llvm

[SYCL][Level Zero] sycl::malloc_device allocates a 1:1 host-side (GTT) mirror when the context contains more than one device

Open
#22,873 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

## Summary

On a multi-GPU Level Zero platform, every `sycl::malloc_device` allocation is accompanied by a
host-side allocation of the **same size**, accounted as GPU-driver memory (GTT). It appears as soon
as the SYCL context contains more than one device, even if only one device is ever used, and even
if nothing is copied between devices.

Allocating via Level Zero directly (`zeMemAllocDevice`) on the same machine, with the same two
devices in the `ze_context`, does **not** produce it. That places the behaviour in the SYCL
runtime's mapping onto L0 rather than in the driver.

The host allocation is invisible to `VmRSS`, `ps` and `free`'s process accounting, which makes it
easy to miss.

## Environment

- 2× Intel Arc Pro B70 (Battlemage / BMG G31)
- Linux 7.1.1 (`xe`), compute-runtime 26.22 / IGC 2.36.3, oneAPI 2026.1
- Reproduced through PyTorch XPU (SYCL backend); no application-specific code involved

## Reproduction

Plain device allocations, ~30 s, no model or framework logic. Only `ZE_AFFINITY_MASK` differs.

```
1 GPU (ZE_AFFINITY_MASK=0), 4 GiB allocated
VmRSS +0.6 MiB
nr_gpu_active +0.00 GiB
0000:0e:00.0 VRAM +4.00 GiB GTT +0.00 GiB

2 GPUs (ZE_AFFINITY_MASK=0,1), 4 GiB allocated on each
VmRSS +5.0 MiB
nr_gpu_active +8.07 GiB
0000:0e:00.0 VRAM +4.00 GiB GTT +4.00 GiB
0000:12:00.0 VRAM +4.18 GiB GTT +4.07 GiB
```

Each device ends up holding its own allocation in VRAM **and** a same-sized GTT entry mirroring the
other device's allocation. Total host cost equals the sum of all device allocations, 1:1. Freeing
the allocations releases it.

Measured with:
- `/proc//fdinfo/` → `drm-total-vram0`, `drm-total-gtt`
- `/sys/devices/system/node/node0/vmstat` → `nr_gpu_active`

`nr_foll_pin_*` moved by 4 pages and `Mlocked` stayed at 3.2 MiB, so this is driver-owned system
memory rather than pinned or locked user pages.

## Level Zero control (same box, SYCL bypassed)

`zeMemAllocDevice` called directly through ctypes, varying only the number of devices in the
`ze_context`:

```
zeMemAllocDevice(4 GiB on device 0) + zeContextMakeMemoryResident

ze_context with 1 device : 0000:0e:00.0 VRAM 4.00 GiB GTT 0.00 GiB nr_gpu_active +0.00 GiB
ze_context with 2 devices: 0000:0e:00.0 VRAM 4.00 GiB GTT 0.00 GiB nr_gpu_active +0.00 GiB
```

No mirror in either case, while the SYCL path on the same hardware produces 8.07 GiB.

*Note:* `zeMemAllocDevice` is lazily committed. Without `zeContextMakeMemoryResident` both cases
report `VRAM 0.00 / GTT 0.00`, which looks like a clean result but means nothing was committed.

## Impact

Frameworks that create one SYCL context spanning all visible devices pay host RAM equal to their
total device allocation. In our case, a tensor-parallel inference server across 2 GPUs consumes host
RAM equal to the total VRAM pool:

```
--gpu-memory-utilization 0.90 -> 57.1 GiB host RAM
0.80 -> 50.7 GiB host RAM
(57.1/50.7 = 1.126 vs 0.90/0.80 = 1.125 — proportional to the VRAM pool, not to model size)
```

The model itself is 45 GiB and resides entirely in VRAM; the host RAM is purely the mirror. On a
123 GiB host that is roughly half the machine, and it contributed to an out-of-memory condition that
was hard to diagnose precisely because the memory does not appear in the usual counters.

Restricting the context to one device removes it, but that is not an option for tensor-parallel
work. `ONEAPI_DEVICE_SELECTOR=level_zero:N` is equivalent to `ZE_AFFINITY_MASK=N` here — it only
removes the second device.

## Related

- llama.cpp issue #22116 reports the same behaviour from a different application, with a proposed
fix of replacing `sycl::malloc_device` with `zeMemAllocDevice` (PR #21597, closed). The Level Zero
control above is consistent with that fix addressing the cause.

Reproducer scripts (SYCL/PyTorch and the raw Level Zero control) can be provided; happy to run
further variants — different allocation sizes, more than two devices, or explicit per-device
contexts — if that helps.

Contributor guide

Open the contributing guide

Research direction

Start at the SYCL runtime path for sycl::malloc_device and compare its behavior with the direct zeMemAllocDevice control described in the issue. Reproduce with a multi-device context, making allocations resident with zeContextMakeMemoryResident, then verify that freeing allocations releases VRAM without creating same-sized GTT mirrors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
49/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.