[610.57.04 / GA102] Mapping-node failures in gpu_vaspace.c (both 4635/2066 and 4551/2021 forms) after sustained Vulkan load pins VRAM at ~96% under Wayland

Summary

On an RTX 3080 (GA102) with nvidia-open-dkms 610.57.04, Deadlock (Source 2)
running its native Vulkan renderer under a Wayland compositor throws repeated
duplicate-mapping-node failures in gpu_vaspace.c after roughly 100 minutes of
sustained load. The GPU does not
hang and no Xid is logged, but frametimes degrade sharply in the same window
and the game process eventually exits.

This is the NV_ERR_INVALID_ARGUMENT / duplicate-node variant, not the
NV_ERR_NO_MEMORY exhaustion variant covered by #1165 and #1140. VRAM peaked at
6.4 GiB of 10 GiB, so nothing was close to full.

Error signature

Three identical bursts, at 102, 103 and 104 minutes into a 105-minute session:

NVRM: nvCheckFailedNoLog: Check failed: 0 == (pMapNode->gpuMask & gpuMask) @ gpu_vaspace.c:4551
NVRM: nvCheckOkFailedNoLog: Check failed: Invalid argument to call [NV_ERR_INVALID_ARGUMENT]
      (0x0000001F) returned from _gvaspaceMappingInsert(pGVAS, pGpu, pVASBlock, vaLo, vaHi, flags)
      @ gpu_vaspace.c:2021
NVRM: nvCheckFailedNoLog: Check failed: NV_OK == status @ virt_mem_allocator_gm107.c:2560
NVRM: dmaAllocMapping_GM107: can't update VA space for mapping @vaddr=0xa200000

The third burst reported @vaddr=0x2a3e00000; the first two both reported
@vaddr=0xa200000, the same address twice.

0 == (pMapNode->gpuMask & gpuMask) failing means the driver found a mapping
node already covering the requested GPU for that VA range, so this is a
duplicate DMA mapping registration rather than an allocation failure.

Environment

GPU	RTX 3080 10 GiB, GA102, 10de:2206 rev a1
Driver	nvidia-open-dkms 610.57.04, open kernel module
Kernel	7.2.3, Arch Linux
Compositor	Hyprland (Wayland)
Application	Deadlock (appid 1422450), Source 2 native Vulkan renderer via -vulkan. Windows build under Proton experimental-11.0-20260910b. DXVK is not in the path
IOMMU	iommu=pt, GPU group domain type identity
PCIe	Gen4 x16, CPU-direct root port, 2-device IOMMU group
ReBAR	active, BAR1 16384 MiB (EnableResizableBar module param reads 0, so this is firmware)
Peak VRAM	6.4 GiB of 10 GiB

Reproduction

Not reliably reproducible on demand. It correlates with session length rather
than with any particular action:

• Sessions of 1.8, 2.8 and 0.6 minutes: clean.
• A 104.7-minute session: three bursts, all after the 102-minute mark.
• A subsequent session was still clean at 55 minutes.

This matches the 30-to-45-minute onset other reporters describe for the
exhaustion variant, but stretched, and without exhaustion being reached.

Measured impact

Frametime percentiles per 10-minute window of the 104.7-minute session, MangoHud
at a 100 ms log interval, first 60 s of load-in dropped:

window	p99 (ms)	p99.9 (ms)	frames >33 ms
first 80 minutes, eight windows	10.2 to 11.9	12 to 33	0 to 5
minutes 80 to 90	13.23	91.29	18
minutes 90 to 100	9.53	12.78	0
minutes 100 to 105	30.54	36.42	7

The final window, which contains all three bursts, has a p99 three times every
clean window's. Median frametime stays flat at about 6 ms throughout, so this is
purely a tail effect.

What was ruled out

• Not thermal. 79 to 81 C, HW Thermal Slowdown and SW Thermal Slowdown
  counters both at 0 us. Only SW Power Cap was active, which is the card at
  its 320 W board limit.

• Not VRAM exhaustion. 6.4 GiB peak of 10 GiB.

• Not DXVK. Source 2 ships rendersystemvulkan.dll, and the game is launched
  with -vulkan so D3D11 translation is out of the path entirely. Verified by
  thread names rather than mapped DLLs, since Source 2 probes and maps both
  render backends at startup either way. Full launch options:

  gamemoderun mangohud __NV_PRIME_RENDER_OFFLOAD=1 VK_LOADER_DRIVERS_SELECT=nvidia_icd.json %command% -vulkan
  

  Worth noting that this game also produced Xid 109 CTX SWITCH TIMEOUT hard
  hangs when it ran through DXVK, which stopped when the renderer was switched.
  Those are a separate failure mode and are not what this report is about; the
  mapping fault below fires under the native Vulkan renderer with no Xid at
  all.

• Not IOMMU translation. iommu=pt was applied specifically to test this and
  the fault survived it unchanged. Domain type reads identity.

• Not PCIe topology. The card was moved from a chipset-attached Gen4 x4 link
  to a CPU-direct Gen4 x16 link, and the fault survived that too.

• No Xid was logged at any point, and the compositor stayed responsive
  throughout, so this is distinct from a GPU hang.

Related

• #1165 and #1140 are the NV_ERR_NO_MEMORY exhaustion variant. Same call path,
  different failure mode.
• The duplicate-node variant appears in one public report I can find, an RTX 5080
  on 575.153.02 under Wayland, with the same 0 == (pMapNode->gpuMask & gpuMask)
  check and the same _gvaspaceMappingInsert rejection:
  https://forums.developer.nvidia.com/t/575-153-02-btree-mapping-crash-during-dune-awakening/336878
  That report received no reply.
• NVIDIA internal ticket #5762513 tracks the dmaAllocMapping_GM107 family
  as a Wayland-only issue, per
  https://forums.developer.nvidia.com/t/bug-report-wayland-only-dmaallocmapping-gm107-va-mapping-failures-does-not-reproduce-on-x11/353598
  That thread lists GA102 among affected hardware and gives an affected range of
  580.105.08 through 610.43.02. This report is on 610.57.04, past that range.

Happy to run instrumented sessions or collect additional logs on request.

Start by reading gpu_vaspace.c around lines 4551 and 2021, then follow the call from virt_mem_allocator_gm107.c:2560 and dmaAllocMapping_GM107. Run a sustained native Vulkan session under Wayland using the reported environment and collect the mapping failures. Done means identifying the duplicate-registration cause and validating a fix without the repeated errors or frametime degradation.