[GSD-13087] UR_RESULT_ERROR_OUT_OF_RESOURCES on 4 GPUs (B70) with large allocations.
- Dominant language
- C++
- Stars
- 1.4k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
### Pre-submission Checklist
- [x] I am using the latest GPU driver version ([releases](https://github.com/intel/compute-runtime/releases))
- [x] I have searched for similar issues and found none
### GPU Hardware
Arc Pro B70
### DRI Devices Information
```
$ ls -la /dev/dri/by-path/
lrwxrwxrwx 1 root root 8 Jul 13 14:51 pci-0000:03:00.0-card -> ../card1
lrwxrwxrwx 1 root root 13 Jul 13 14:51 pci-0000:03:00.0-render -> ../renderD129
lrwxrwxrwx 1 root root 8 Jul 13 14:51 pci-0000:23:00.0-card -> ../card2
lrwxrwxrwx 1 root root 13 Jul 13 14:51 pci-0000:23:00.0-render -> ../renderD130
lrwxrwxrwx 1 root root 8 Jul 13 14:51 pci-0000:c3:00.0-card -> ../card3
lrwxrwxrwx 1 root root 13 Jul 13 14:51 pci-0000:c3:00.0-render -> ../renderD131
lrwxrwxrwx 1 root root 8 Jul 13 14:51 pci-0000:f3:00.0-card -> ../card0
lrwxrwxrwx 1 root root 13 Jul 13 14:51 pci-0000:f3:00.0-render -> ../renderD128
```
### GPU Detailed Information (lspci output)
```
$ for c in /dev/dri/by-path/pci-*-card; do a=${c##*/pci-}; sudo lspci -vvv -k -s "${a%-card}"; done
03:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00004 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000014000000000 (64-bit, prefetchable)
Region 2: Memory at 0000013800000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: xe
Kernel modules: xe
23:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00004 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000015800000000 (64-bit, prefetchable)
Region 2: Memory at 0000015000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: xe
Kernel modules: xe
c3:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00004 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000012800000000 (64-bit, prefetchable)
Region 2: Memory at 0000012000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: xe
Kernel modules: xe
f3:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 1701
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00004 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000011000000000 (64-bit, prefetchable)
Region 2: Memory at 0000010800000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: xe
Kernel modules: xe
```
### Driver Version
26.22.38646.6-1~24.04~ppa1
### Installed GPU Driver Packages
```
$ sudo dpkg --list | grep -iE "igc|gmm|opencl|level-zero|fc|level_zero|ocloc|libze"
ii bpfcc-tools 0.29.1+ds-1ubuntu7 all tools for BPF Compiler Collection (BCC)
ii clinfo 3.0.23.01.25-1build1 amd64 Query OpenCL system information
ii intel-igc-core-2 2.34.4 amd64 Intel(R) Graphics Compiler for OpenCL(TM)
ii intel-igc-opencl-2 2.34.4 amd64 Intel(R) Graphics Compiler for OpenCL(TM)
ii intel-ocloc 26.22.38646.6-1~24.04~ppa1 amd64 Tool for managing Intel Compute GPU device binary format
ii intel-opencl-icd 26.22.38646.6-1~24.04~ppa1 amd64 Intel graphics compute runtime for OpenCL
ri level-zero 1.28.2 amd64 oneAPI Level Zero
ii level-zero-devel 1.28.2 amd64 oneAPI Level Zero
ii libbpfcc:amd64 0.29.1+ds-1ubuntu7 amd64 shared library for BPF Compiler Collection (BCC)
ii libcbor0.10:amd64 0.10.2-1.2ubuntu2 amd64 library for parsing and generating CBOR (RFC 7049)
ii libdebconfclient0:amd64 0.271ubuntu3 amd64 Debian Configuration Management System (C-implementation library)
ii libfile-fcntllock-perl 0.22-4ubuntu5 amd64 Perl module for file locking with fcntl(2)
ii libigc2 2.36.3-2~24.04 amd64 Core libraries for Intel(R) Graphics Compiler for OpenCL(TM)
ii libigdfcl2 2.36.3-2~24.04 amd64 OpenCL library for Intel(R) Graphics Compiler for OpenCL(TM)
ii libigdgmm12:amd64 22.10.0-1~24.04~ppa1 amd64 Intel Graphics Memory Management Library -- shared library
ii libref-array1t64:amd64 0.6.2-2.1build1 amd64 refcounted array for C
ii libsigc++-3.0-0:amd64 3.6.0-2 amd64 type-safe Signal Framework for C++ - runtime
ii libze-intel-gpu1 26.22.38646.6-1~24.04~ppa1 amd64 Intel oneAPI L0 support implementation for Intel GPUs -- shared library
ii libze1:amd64 1.28.6-1~24.04~ppa1 amd64 oneAPI Level Zero -- share libraries
ii ocl-icd-libopencl1:amd64 2.3.2-1build1 amd64 Generic OpenCL ICD Loader
ii python3-bpfcc 0.29.1+ds-1ubuntu7 all Python 3 wrappers for BPF Compiler Collection (BCC)
ii python3-idna 3.6-2ubuntu0.1 all Python IDNA2008 (RFC 5891) handling (Python 3)
```
### Driver Installation Details
Installed through `apt` on Ubuntu 24.04.
### Linux Distribution
Ubuntu 24.04 LTS
### Other Linux Distribution
_No response_
### Kernel Version & Boot Parameters
```
$ uname -r
6.17.0-35-generic
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.17.0-35-generic root=UUID=cda26a52-4206-48e9-9beb-ed6d68441b9f ro splash vt.handoff=7
$ lsmod | grep -E 'i915|xe'
xe 3874816 127
intel_vsec 24576 2 pmt_telemetry,xe
gpu_sched 65536 1 xe
drm_gpuvm 49152 1 xe
drm_gpusvm_helper 36864 1 xe
drm_buddy 28672 1 xe
drm_ttm_helper 16384 1 xe
ttm 126976 2 drm_ttm_helper,xe
drm_exec 12288 2 drm_gpuvm,xe
drm_suballoc_helper 24576 1 xe
drm_display_helper 290816 1 xe
cec 98304 2 drm_display_helper,xe
i2c_algo_bit 16384 1 xe
video 77824 2 asus_wmi,xe
```
### Actual Behavior
When using large allocations, the driver ends up reporting `UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY` on 4 GPUs (B70), but not on 2. The issue seems very similar to this one: https://github.com/intel/compute-runtime/issues/952, which was closed because the author thinks it's the same underlying issue as https://github.com/intel/compute-runtime/issues/953. It's unclear to me whether it's really the same problem, I couldn't clearly identify any metric on my kernel (6.17.0-35-generic) / xe driver that clearly indicated that host memory was the problem. However, disabling peer residency and avoiding P2P memcpy solves the issue for me.
Using `NEOReadDebugKeys=1 EnableDeferBacking=0` as suggested in the https://github.com/intel/compute-runtime/issues/953 had no effect. I'll keep following the other issue.
### Expected Behavior
The driver shouldn't fail with out of memory errors just for allocating huge memory chunks on each device.
### Reproduction Rate
Always reproduces - 100%
### Steps to Reproduce
I have two reproductions in pure SYCL:
- The first one uses a single context and allocates a huge slab in each of the devices' memory. This fails for the third device. However, disabling peer residency with `UR_LOADER_USE_LEVEL_ZERO_V2=0 SYCL_UR_USE_LEVEL_ZERO_V2=0 UR_L0_USM_RESIDENT=0x001` works.
- The second one uses a different context for each device. It manages to make the huge allocations, but fails during P2P memcpy.
Single context reproduction
```cpp
#include
#include
#include
int main() {
std::vector gpus;
for (const sycl::platform &platform : sycl::platform::get_platforms())
if (platform.get_backend() == sycl::backend::ext_oneapi_level_zero)
for (const sycl::device &device : platform.get_devices())
if (device.is_gpu())
gpus.push_back(device);
if (gpus.size() < 4) {
std::cerr << "need 4 Level Zero GPUs, found " << gpus.size() << "\n";
return 2;
}
constexpr std::size_t num_gpus = 4;
sycl::context context({gpus[0], gpus[1], gpus[2], gpus[3]});
std::vector queues;
std::vector slabs(num_gpus);
std::vector slab_bytes(num_gpus);
queues.reserve(num_gpus);
for (std::size_t i = 0; i < num_gpus; ++i) {
queues.emplace_back(context, gpus[i], sycl::property::queue::in_order{});
}
for (std::size_t i = 0; i < num_gpus; ++i) {
slab_bytes[i] =
gpus[i].get_info() * 9 / 10;
slabs[i] = sycl::malloc_device(slab_bytes[i], queues[i]);
std::cout << "slab[" << i << "]=" << slabs[i] << " bytes=" << slab_bytes[i]
<< "\n";
if (!slabs[i])
return 1;
}
}
```
Multiple contexts reproduction
```cpp
#include
#include
#include
int main() {
std::vector gpus;
for (const sycl::platform &platform : sycl::platform::get_platforms())
if (platform.get_backend() == sycl::backend::ext_oneapi_level_zero)
for (const sycl::device &device : platform.get_devices())
if (device.is_gpu())
gpus.push_back(device);
if (gpus.size() < 4) {
std::cerr << "need 4 Level Zero GPUs, found " << gpus.size() << "\n";
return 2;
}
constexpr std::size_t num_gpus = 4;
std::vector contexts;
std::vector queues;
std::vector slabs(num_gpus);
std::vector slab_bytes(num_gpus);
contexts.reserve(num_gpus);
queues.reserve(num_gpus);
for (std::size_t i = 0; i < num_gpus; ++i) {
contexts.emplace_back(gpus[i]);
queues.emplace_back(contexts.back(), gpus[i],
sycl::property::queue::in_order{});
}
for (std::size_t i = 0; i < num_gpus; ++i) {
slab_bytes[i] =
gpus[i].get_info() * 9 / 10;
slabs[i] = sycl::malloc_device(slab_bytes[i], queues[i]);
std::cout << "slab[" << i << "]=" << slabs[i] << " bytes=" << slab_bytes[i]
<< "\n";
if (!slabs[i])
return 1;
}
constexpr std::size_t copy_bytes = 1024;
for (std::size_t src = 0; src < num_gpus; ++src) {
for (std::size_t dst = 0; dst < num_gpus; ++dst) {
if (src == dst)
continue;
queues[dst].memcpy(slabs[dst], slabs[src], copy_bytes).wait_and_throw();
std::cout << "copied " << copy_bytes << " bytes from device " << src
<< " to device " << dst << "\n";
}
}
}
```
### Is this a regression?
- [ ] Yes, this is a regression - functionality that previously worked is now broken
### Last Known Working Driver Version
_No response_
### First Known Failing Driver Version
_No response_
### API Call Logs
_No response_
### strace Logs
_No response_
### System Logs / dmesg Output
_No response_
### Backtrace (if crash or hang occurred)
_No response_
### Source Code / Reproducer
_No response_
### Command Line / Application Details
_No response_
### oneAPI Version (if applicable)
```
$ icpx --version
Intel(R) oneAPI DPC++/C++ Compiler 2026.0.0 (2026.0.0.20260331)
Target: x86_64-unknown-linux-gnu
Thread model: posix
```
### Screenshots / Video
_No response_
### Additional Notes
_No response_
Contributor guide
Research direction
No source file, test, or code entry point is named. Start by reproducing the UR_RESULT_ERROR_OUT_OF_RESOURCES case with four Arc Pro B70 GPUs and large allocations, then trace the failure through the runtime's allocation path. Done means the multi-GPU allocation behavior is explained and covered by a focused regression test or a clearly documented limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100