Misleading -ENOMEM errors caused by nvidia-drm on multi-GPU system

NVIDIA Open GPU Kernel Modules Version

610.57.04

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• [x ] I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Arch Linux

Kernel Release

7.1.0-2-cachyos

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA GeForce RTX 5090

Describe the bug

My desktop is primarily composited by a secondary amdgpu driver, but I offload much my graphics rendering to my 5090. The vivmem-backend buffer is forced through nvidia-drm PRIME, but nv_drm_gem_prime_get_sg_table() returns ERR_PTR(-ENOMEM), which causes my AMD card to think it's out of memory and crashes kwin_wayland.

To Reproduce

• Dual-GPU system: Wayland desktop composited by a non-NVIDIA GPU, NVIDIA GPU present as a render device
• An application on the NVIDIA GPU whose shared surfaces are allocated in video memory rather than through the PRIME-aware sysmem path.

An example: The Windows Steam client's steamwebhelper crashes on every launch. It runs GPU-accelerated Chromium/CEF (D3D11 via Wine + DXVK picking the NVIDIA adapter). It might seem weird that I'm trying to run Windows-native Steam under Wine, but it's just a current project I'm working on. Ordinary Vulkan swapchains take the PRIME sysmem path and thus don't trigger any crashes

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

Proposed fix: give objects that can never produce an sg_table a funcs table without .get_sg_table. This is chosen automatically for GEM types lacking a prime_get_sg_table op, and per-instance for vidmem-backed NVKMS memory.

I also wrote a patch to help fix this (sorry for all the comments; I did ask AI to help me).

nvidia-drm-vidmem-no-sgt.patch

Start by tracing nv_drm_gem_prime_get_sg_table() and reviewing the attached nvidia-drm-vidmem-no-sgt.patch. Compare the proposed GEM funcs-table handling with the existing PRIME and vidmem-backed paths. Done means video-memory objects no longer return misleading -ENOMEM through nvidia-drm, and the described dual-GPU Wayland reproduction no longer crashes its compositor.