NVIDIA / NVIDIA/open-gpu-kernel-modules

mmuWalkUnmap failure → "GPU lost from the bus" → UVM fatal error 0x60 during CUDA generation (RTX 5090, driver 610.57.04)

Open
#1,342 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
17.4k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

NVIDIA Open GPU Kernel Modules Version

610.57.04 (open kernel module)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
  • I confirm that this does not happen with the proprietary driver package.
Operating System and Version

Arch linux

Kernel Release

7.2.3-arch1-2

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
  • I am running on a stable kernel release.
Hardware: GPU

RTX 5090 (Blackwell)

Describe the bug
  • Trigger: crashed during a PyTorch generate() call (LLM inference, KV-cache alloc/free churn) shortly after model load completed on GPU; not under any virtualization/passthrough
  • Symptom: nvidia-smi starts reporting Unable to determine the device handle for GPU0: 0000:09:00.0: Unknown Error / No devices were found; only a full reboot recovers it
  • Root chain from dmesg (attached in full): mmuWalkUnmap: Failed to unmap VA Range ... Status = 0x00000040 → repeated kgmmuInvalidateTlb_GM107: TLB invalidation failed → GPU lost from the bus [NV_ERR_GPU_IS_LOST] → nvGpuOpsReportFatalError: uvm encountered global fatal error 0x60, requiring os reboot to recover → cascading GspRmFree failed RPC timeouts → nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress repeating every 5s for 5+ minutes → display-stack processes (e.g. WebKitWebProces) hang indefinitely on an nvkms_close semaphore
Related reports:
  • #916
  • basecamp/omarchy#5554

Reporting as a third data point since the trigger here is neither VM passthrough nor hibernate, just ordinary CUDA inference.

  • Full dmesg attached

dmseg.log

To Reproduce

na

Bug Incidence

Always

nvidia-bug-report.log.gz

.

More Info

.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached dmesg and trace the reported mmuWalkUnmap failure through kgmmuInvalidateTlb_GM107, GPU loss, and UVM fatal error 0x60. Use the PyTorch generate() workload and RTX 5090 environment described here to investigate; done requires a reproducible diagnosis and a validated driver fix, but no reproduction steps or test entry point are provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux, pytorch
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.