NVIDIA / NVIDIA/open-gpu-kernel-modules

595.71.05: list_del corruption storm in nv_mem_pool_scr kthread → slab corruption spreads to dm-crypt → kernel panic (RTX 5090 Laptop / Blackwell)

Open
#1,245 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
17.4k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

NVIDIA Open GPU Kernel Modules Version

595.71.05 (release tag) — Ubuntu 26.04 pre-built signed modules: linux-modules-nvidia-595-open-7.0.0-27-generic 7.0.0-27.27, built from nvidia-kernel-source-595-open 595.71.05-0ubuntu0.26.04.1; modinfo srcversion 58D233B8E3F4A2973D73151 (not a git build, so no SHA)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.
  • I confirm that this does not happen with the proprietary driver package.
Operating System and Version

Ubuntu 26.04 LTS

Kernel Release

Linux asus-ROG 7.0.0-27-generic #27-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 18 19:13:49 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.
  • I am running on a stable kernel release.
Hardware: GPU

GPU 0: NVIDIA GeForce RTX 5090 Laptop GPU

Describe the bug

Roughly once a day during ordinary desktop use (GNOME/Wayland, browser — no CUDA, no
gaming), the kernel starts emitting a storm of list_del corruption warnings from the
driver's memory-pool scrubber kthread nv_mem_pool_scr, and seconds later the slab
corruption spreads to an unrelated subsystem and the machine panics or the root disk
I/O dies.

Captured crash (2026-07-16 19:08, via EFI pstore with crash_kexec_post_notifiers):

1. First warning — nv_mem_pool_scr (PID 1855) walking its own pool list:

slab kmalloc-rnd-13-32 start ffff8e452c1e73e0 pointer offset 8 size 32
list_del corruption. next->prev should be ffff8e452c1e7ba8, but was ffff8e3a2c1e7ba8. (next=ffff8e452c1e73e8)
WARNING: lib/list_debug.c:65 at __list_del_entry_valid_or_report+0xc0/0x10b, CPU#1: nv_mem_pool_scr/1855

(the first bad pointer differs from the expected value in a single byte)

2. 185 further list_del corruption warnings over the next 18 seconds, all in
nv_mem_pool_scr, same slab cache (kmalloc-rnd-13-32). Registers hold LIST_POISON
values (R14: dead000000000122 R15: dead000000000100) — entries that were already
unlinked are being unlinked again (double-unlink / use-after-free signature). Every
warning has the same call trace:

Call Trace:
 <TASK>
 nv_mem_pool_worker.cold+0x8/0x1d [nvidia]
 _main_loop+0x99/0x160 [nvidia]
 ? __pfx__main_loop+0x10/0x10 [nvidia]
 kthread+0xf7/0x130
 ret_from_fork+0x195/0x2a0
 ret_from_fork_asm+0x1a/0x30
 </TASK>

3. 18 seconds after the first warning, the corruption hits the dm-crypt writer
thread for the (LUKS-encrypted) root disk, which faults on a garbage pointer and
panics the machine:

Oops: general protection fault, probably for non-canonical address 0x5c420d97e5ea5cd2: 0000 [#1] SMP NOPTI
CPU: 18 UID: 0 PID: 1208 Comm: dmcrypt_write/2 Kdump: loaded Tainted: G     U  W  O        7.0.0-27-generic #27-Ubuntu PREEMPT(lazy)
Hardware name: ASUSTeK COMPUTER INC. ROG Strix SCAR 18 G835LX_G835LX/G835LX, BIOS G835LX.333 01/23/2026
RIP: 0010:bdev_count_inflight_rw.part.0+0x4c/0x130
Call Trace:
 <TASK>
 bdev_count_inflight+0x22/0x50
 update_io_ticks+0xbf/0xe0
 bdev_start_io_acct+0x2c/0xb0
 dm_io_acct+0x120/0x160
 dm_start_io_acct+0x3c/0xb0
 dm_submit_bio_remap+0x36/0xe0
 dmcrypt_write+0x129/0x160 [dm_crypt]
 kthread+0xf7/0x130
 ret_from_fork+0x195/0x2a0
 ret_from_fork_asm+0x1a/0x30
 </TASK>

Additional context:

  • Earlier crashes on this machine showed the root disk flipping read-only instead of
    panicking — consistent with the same slab corruption landing on the block layer
    without an immediate oops. On other occasions the machine hard-froze with no trace.
  • Same behavior was seen on Ubuntu 25.10 with 580-open before upgrading, so the issue
    spans at least the 580 → 595 open branches.
  • The same hardware is fully stable under Windows, including sustained gaming loads
    (argues against RAM/hardware).
  • Ruled out already (crashes persisted through all of these): pcie_aspm=off, NVMe
    APST disabled, NVIDIA runtime PM disabled (NVreg_DynamicPowerManagement=0x00),
    NVMe PCIe link pinned Gen5→Gen4, intel_idle.max_cstate=2, distro upgrade
    25.10 → 26.04 (kernel 6.17 → 7.0, driver 580-open → 595-open).
  • Note re: proprietary driver comparison: this GPU is Blackwell, which is supported
    only by the open kernel modules, so a proprietary-driver comparison is not possible
    on this hardware. The crashing kthread (nv_mem_pool_scr / nv_mem_pool_worker) is
    kernel-module-side code from this repo.
  • Crash occurs ~once/day, so I can test candidate fixes or a DKMS build from any
    commit with ~1–2 day turnaround.
To Reproduce

There is no known deterministic trigger — the corruption starts spontaneously during
ordinary desktop use. Environment/preconditions when it hits:

  • ASUS ROG Strix SCAR 18 (G835LX), RTX 5090 Laptop GPU (hybrid graphics with Intel
    Arrow Lake HX iGPU), Ubuntu 26.04, kernel 7.0.0-27-generic, driver 595.71.05-open
    (Ubuntu packaged)
  • GNOME Wayland session, typically with Microsoft Edge running; Docker daemon with
    ~40 containers active in the background (CPU-only, no GPU containers, no CUDA)
  • No gaming / no deliberate GPU load — light desktop use or near-idle
  • Runtime PM disabled (NVreg_DynamicPowerManagement=0x00); no suspend/resume cycle
    involved — crashes occur mid-session hours after boot (captured one was ~3h8m uptime)

Observed sequence when it reproduces:

  • nv_mem_pool_scr begins emitting list_del corruption warnings
    (lib/list_debug.c:65) against slab kmalloc-rnd-13-32
  • warnings repeat (185 times over 18 s in the captured instance)
  • slab corruption then hits an unrelated kernel subsystem → panic (captured instance:
    GPF in bdev_count_inflight_rw from dmcrypt_write), or on other occasions the
    root disk I/O path dies and the FS remounts read-only, or the machine hard-freezes
Bug Incidence

Sometimes

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

pstore-crash-2026-07-16-part1.txt
pstore-crash-2026-07-16-part2.txt

journal-warning-storm-2026-07-16.txt

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the nv_mem_pool_scr worker and its nv_mem_pool_worker/_main_loop call trace, then correlate the attached pstore and journal logs with the repeated list_del warnings. There is no deterministic reproducer or source file named; done would require isolating the corruption, reproducing it reliably, and verifying that the warning storm and resulting kernel panic no longer occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.