[GB202 / RTX 5090] Uninterruptible D-state deadlock in RemoteDpcdReadMessage / MessageManager::send on DisplayPort modeset with MST branch device

NVIDIA Open GPU Kernel Modules Version

610.57.04 (and 595.84) - linux-modules-nvidia-610-open-generic / nvidia-driver-610-open

Please confirm this issue does not happen with the proprietary driver

• I confirm that this does not happen with the proprietary driver package.
  (Note: RTX 50 series / GB202 Blackwell only supports the open kernel modules; the legacy proprietary closed driver cannot be used).

Operating System and Version

Ubuntu 26.04 LTS

Kernel Release

Linux medina 7.0.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug 1 04:26:38 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-96639ba1-6655-f088-8ffb-e025ec620129, Device ID: 2B85)

Describe the bug

During an atomic modeset commit under GNOME Wayland (Mutter), specifically when entering screen blanking or renegotiating DisplayPort links with an MST-enabled display, the driver enters an uninterruptible sleep (D state) semaphore deadlock.

The kernel begins reporting hung tasks after 122 seconds:
INFO: task nvidia-modeset/:1508 blocked on a semaphore likely last held by task KMS thread:16987

Once this deadlock occurs:

1. nvidia-modeset/kthread_q is trapped in D state and cannot be killed.
2. KMS thread (gnome-shell) is trapped in D state inside RemoteDpcdReadMessage / MessageManager::send.
3. The kernel module nvidia_modeset cannot be unloaded.
4. All subsequent atomic commits fail with Page flip failed: drmModeAtomicCommit: Invalid argument.
5. DisplayPort monitor fails to wake; only a hard system reboot can clear the deadlock.

Stack Trace - Thread 1 (nvidia-modeset/kthread_q, PID 1508):

task:nvidia-modeset/ state:D stack:0 pid:1508
Call Trace:
 <TASK>
 __schedule+0x2bb/0x650
 schedule+0x27/0xb0
 schedule_timeout+0xcf/0x110
 ___down_common+0x108/0x1e0
 __down_common+0x5a/0x180
 __down+0x1d/0x30
 down+0x5e/0x80
 nvkms_kthread_q_callback+0xb4/0x190 [nvidia_modeset]
 _main_loop+0x99/0x160 [nvidia_modeset]
 kthread+0xf7/0x130
 ret_from_fork+0x195/0x2a0
 ret_from_fork_asm+0x1a/0x30
 </TASK>

Stack Trace - Thread 2 (KMS thread, PID 16987, holding semaphore and waiting on DP AUX):

task:KMS thread state:D stack:0 pid:16987 tgid:16972
Call Trace:
 <TASK>
 __schedule+0x2bb/0x650
 schedule+0x27/0xb0
 schedule_timeout+0x88/0x110
 schedule_timeout_uninterruptible+0x29/0x50
 msleep+0x1b/0x30
 nvkms_usleep+0x4f/0x60 [nvidia_modeset]
 _ZN11DisplayPort14MessageManager4sendEPNS0_7MessageERNS_7NakDataE+0xf7/0x2f0 [nvidia_modeset]
 _ZN11DisplayPort21RemoteDpcdReadMessage3setERKNS_7AddressEjjj+0xca/0x150 [nvidia_modeset]
 _ZN11DisplayPort10DeviceImpl11transactionENS_6AuxBus6ActionENS1_4TypeEiPhjPjS5_hh+0xc21/0x12f0 [nvidia_modeset]
 _ZN11DisplayPort10DeviceImpl11getDpcdDataEjPhjPjS2_+0xb3/0x170 [nvidia_modeset]
 _ZN11DisplayPort10DeviceImpl18setIgnoreMSAEnableEb+0x95/0x130 [nvidia_modeset]
 nvDPLibSetAdaptiveSync+0x66/0x90 [nvidia_modeset]
 nvDisableVrr+0x298/0x350 [nvidia_modeset]
 nvSetDispModeEvo+0x2e7a/0x40a0 [nvidia_modeset]
 nvKmsIoctl+0xf2/0x240 [nvidia_modeset]
 nvkms_ioctl_from_kapi_try_pmlock+0x6b/0xc0 [nvidia_modeset]
 ApplyModeSetConfig+0x382/0xc70 [nvidia_modeset]
 nv_drm_atomic_apply_modeset_config+0x2c9/0xb40 [nvidia_drm]
 nv_drm_atomic_commit+0x25c/0x5d0 [nvidia_drm]
 drm_atomic_commit+0xad/0xf0
 drm_mode_atomic_ioctl+0x7d1/0x910
 </TASK>

To Reproduce

1. Connect a display with an internal MST hub via DisplayPort (Dell U3225QE) alongside a secondary display (Dell U2719DC on HDMI) to an RTX 5090.
2. Run GNOME Wayland session.
3. Have CUDA compute workload resident in background (e.g. Ollama with ~24 GB allocated in VRAM).
4. Trigger screen-blank modeset (e.g. idle timeout idle-delay 300 or DP link renegotiation).
5. The display fails to wake from blank; nvidia-modeset enters uninterruptible sleep D state with hung task reports repeating every 122 seconds.

Bug Incidence

Sometimes (reproduced on multiple days under both driver 595.84 and 610.57.04).

nvidia-bug-report.log.gz

(Please find nvidia-bug-report.log.gz attached to this issue, captured via sudo nvidia-bug-report.sh --safe-mode during the active deadlock).

More Info

Notice that nvSetDispModeEvo invokes setIgnoreMSAEnable, which issues RemoteDpcdReadMessage to the MST branch device. The modeset semaphore is held across the wait inside MessageManager::send (nvkms_usleep / schedule_timeout_uninterruptible). If the DP AUX reply is lost or delayed, the timeout mechanism appears unable to release the lock, permanently hanging both threads and preventing any future modesets until a hard reboot.

Begin with the reported atomic commit path at nv_drm_atomic_commit and the nvidia_modeset entry points nvSetDispModeEvo, setIgnoreMSAEnable, RemoteDpcdReadMessage, and MessageManager::send; reproduce with the listed MST/HDMI setup and collect the deadlock stacks. Done means blanking or DP renegotiation completes without an uninterruptible semaphore deadlock, hung tasks, or requiring a reboot.