intel / intel/compute-runtime

[GSD-13095] SYCL global_mem_size reports PF total LMEM instead of VF partition on some SR-IOV VFs (Arc Pro B70)

Open
#959 1 comment 0 reactions 0 assignees View on GitHub
OS: Linux Status: Needs Feedback Type: Bug
Dominant language
C++
Stars
1.4k
Forks
300
PR merge metrics
No merged PRs in 30d

Description

### Pre-submission Checklist

- [x] I am using the latest GPU driver version ([releases](https://github.com/intel/compute-runtime/releases))
- [x] I have searched for similar issues and found none

### GPU Hardware

Intel Arc Pro B70 (BMG-G31, PCI ID 8086:e223)

### DRI Devices Information

2x Arc Pro B70 PFs with 4 SR-IOV VFs each (8 VFs total):

```
$ ls -la /dev/dri/by-path/
pci-0000:18:00.0-card -> ../card1 # GPU1 PF
pci-0000:18:00.0-render -> ../renderD128
pci-0000:18:00.1-card -> ../card3 # GPU1 VF1
pci-0000:18:00.1-render -> ../renderD130
pci-0000:18:00.2-card -> ../card4 # GPU1 VF2
pci-0000:18:00.2-render -> ../renderD131
pci-0000:18:00.3-card -> ../card5 # GPU1 VF3
pci-0000:18:00.3-render -> ../renderD132
pci-0000:18:00.4-card -> ../card6 # GPU1 VF4
pci-0000:18:00.4-render -> ../renderD133
pci-0000:54:00.0-card -> ../card2 # GPU2 PF
pci-0000:54:00.0-render -> ../renderD129
pci-0000:54:00.1-card -> ../card7 # GPU2 VF1
pci-0000:54:00.1-render -> ../renderD134
pci-0000:54:00.2-card -> ../card8 # GPU2 VF2
pci-0000:54:00.2-render -> ../renderD135
pci-0000:54:00.3-card -> ../card9 # GPU2 VF3
pci-0000:54:00.3-render -> ../renderD136
pci-0000:54:00.4-card -> ../card10 # GPU2 VF4
pci-0000:54:00.4-render -> ../renderD137
```

### GPU Detailed Information (lspci output)

```
18:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Region 0: Memory at 207804000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at 206800000000 (64-bit, prefetchable) [size=32G]
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
Initial VFs: 4, Total VFs: 4, Number of VFs: 4, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Kernel driver in use: xe
```

All VF BAR2 sizes are identical at 8 GiB (`stat -c %s /sys/bus/pci/devices/0000:XX:00.Y/resource2` = 8589934592 for all VFs).

### Driver Version

26.18.38308.4

### Installed GPU Driver Packages

Tested inside `docker.io/intel/oneapi:2026.0.0-devel-rockylinux10` container with packages upgraded to latest:

```
intel-compute-runtime-26.18.38308.4-1.el10_2.x86_64
intel-level-zero-26.18.38308.4-1.el10_2.x86_64
intel-igc-2.34.4-4.el10_2.x86_64
intel-igc-libs-2.34.4-4.el10_2.x86_64
intel-gmmlib-22.8.1-2.el10_1.x86_64
intel-opencl-26.18.38308.4-1.el10_2.x86_64
intel-ocloc-26.18.38308.4-1.el10_2.x86_64
```

### Driver Installation Details

Packages installed from the Rocky Linux 10 Intel GPU repos (`dnf upgrade intel-level-zero intel-compute-runtime intel-igc intel-igc-libs intel-opencl intel-ocloc oneapi-level-zero`). Workloads run as Kubernetes pods with DRA-assigned `/dev/dri/renderD*` devices (one VF per pod).

### Linux Distribution

Fedora / RHEL

### Other Linux Distribution

RHEL CoreOS 9.8 (OpenShift 4.22.2). Userspace runs inside Rocky Linux 10 containers.

### Kernel Version & Boot Parameters

```
$ uname -r
5.14.0-687.15.1.el9_8.x86_64

$ cat /proc/cmdline
... pci=realloc intel_iommu=on iommu=pt ...

$ lsmod | grep xe
xe 3985408 4
```

### Actual Behavior

`sycl::info::device::global_mem_size` reports the full PF LMEM size (~31023 MiB) on some SR-IOV VFs instead of the correct per-VF partition (~7719 MiB). Affected VFs vary per GPU — on this system, GPU1 VF1 and VF4 are affected while GPU1 VF2-3 and all GPU2 VFs report correctly.

Tested by deploying one SYCL pod per VF (PCI-address-pinned via DRA ResourceClaims), each running:

```cpp
auto dev = q.get_device();
std::cout << "Memory: " << dev.get_info() / (1024*1024) << " MiB" << std::endl;
```

Results:

| VF | PCI Address | SYCL global_mem_size | Expected |
|---|---|---|---|
| GPU1 VF1 | `0000:18:00.1` | **31023 MiB** | 7719 MiB |
| GPU1 VF2 | `0000:18:00.2` | 7719 MiB | 7719 MiB |
| GPU1 VF3 | `0000:18:00.3` | 7719 MiB | 7719 MiB |
| GPU1 VF4 | `0000:18:00.4` | **31023 MiB** | 7719 MiB |
| GPU2 VF1 | `0000:54:00.1` | 7719 MiB | 7719 MiB |
| GPU2 VF2 | `0000:54:00.2` | 7719 MiB | 7719 MiB |
| GPU2 VF3 | `0000:54:00.3` | 7719 MiB | 7719 MiB |
| GPU2 VF4 | `0000:54:00.4` | 7719 MiB | 7719 MiB |

The values 7719 and 31023 are consistent with a 0.95 headroom factor applied to the correct VF size (8126 MiB) and the PF visible size (32656 MiB), respectively: `8126 * 0.95 ≈ 7719`, `32656 * 0.95 ≈ 31023`.

**The kernel is not the source.** The xe driver's TTM resource manager reports the correct size for all VFs, confirmed via debugfs:

```
# cat /sys/kernel/debug/dri/{card}/vram0_mm | grep 'man size'
card1 (0000:18:00.0) [PF]: man size:34242297856 (32656 MiB)
card3 (0000:18:00.1) [VF]: man size:8520728576 (8126 MiB) <-- correct
card4 (0000:18:00.2) [VF]: man size:8520728576 (8126 MiB)
card5 (0000:18:00.3) [VF]: man size:8520728576 (8126 MiB)
card6 (0000:18:00.4) [VF]: man size:8520728576 (8126 MiB) <-- correct
card7 (0000:54:00.1) [VF]: man size:8520728576 (8126 MiB)
...all VFs identical...
```

The `DRM_IOCTL_XE_DEVICE_QUERY` (MEM_REGIONS) response from the kernel is correct for all VFs. The misreporting happens in the compute-runtime/L0 userspace layer.

dmesg also confirms correct VF LMEM provisioning by the PF:

```
xe 0000:18:00.0: [drm] Tile0: GT0: PF: VF1..VF4 provisioned with 8520728576 (7.94 GiB) LMEM
xe 0000:54:00.0: [drm] Tile0: GT0: PF: VF1..VF4 provisioned with 8520728576 (7.94 GiB) LMEM
```

### Expected Behavior

`sycl::info::device::global_mem_size` should report the VF's actual LMEM partition (~7719 MiB after headroom) on all VFs, matching the kernel-reported `man->size` of 8520728576 bytes.

### Reproduction Rate

Always reproduces - 100%

### Steps to Reproduce

1. System with 2x Intel Arc Pro B70 GPUs, xe driver loaded
2. Create 4 SR-IOV VFs per GPU: `echo 4 > /sys/bus/pci/devices/0000:XX:00.0/sriov_numvfs`
3. Run a SYCL program that queries `sycl::info::device::global_mem_size` on each VF independently (one process per VF, each seeing only its own `/dev/dri/renderD*`)
4. Compare reported memory across VFs — some report ~31023 MiB (PF value) instead of ~7719 MiB (VF value)

### oneAPI Version (if applicable)

```
$ icpx --version
Intel(R) oneAPI DPC++/C++ Compiler 2026.0.0 (2026.0.0.20260331)
Target: x86_64-unknown-linux-gnu
```

### Additional Notes

- Compute performance is **not affected** — all VFs deliver identical GFLOPS regardless of the reported memory value. The actual LMEM allocation and compute work correctly; only the reporting is wrong.
- vLLM inference also works correctly on affected VFs — `Free memory on device (7.03/7.54 GiB)` is reported accurately by the vLLM memory profiler path, which uses `torch.xpu.mem_get_info()` rather than `global_mem_size`.
- The pattern of which VFs are affected is not consistent with a "first VF" or "last VF" rule — it's VF1 + VF4 on GPU1 only, while GPU2 is unaffected.

Contributor guide

Open the contributing guide

Research direction

Start at the SYCL device-info query for global_mem_size in the compute-runtime/Level Zero userspace path, then compare its input with the DRM_IOCTL_XE_DEVICE_QUERY MEM_REGIONS response and the xe debugfs vram0_mm values described here. Reproduce across the listed SR-IOV VFs and verify that every VF reports about 7719 MiB rather than the PF value of about 31023 MiB.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.