NVIDIA / NVIDIA/open-gpu-kernel-modules

GPU register reads stall all CPU cores ~4-5 ms; mem-clock switches cause stall storms and audio dropouts (610.43.03, RTX 5080 Mobile, GSP)

Open
#1,248 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
17.4k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

Summary

nvidia-powerd (Dynamic Boost) causes synchronized ~4–5 ms stalls of all CPU cores simultaneously, 2–3 times per second, while a game is rendering on the dGPU. Each stall is long enough to overrun typical PipeWire audio buffers (5–10 ms), producing frequent audible audio dropouts on any output device (verified on both an ALSA analog sink and a Bluetooth sink). Stopping nvidia-powerd reduces the stall rate ~5×, and the audio dropouts become unnoticeable; starting it brings them straight back (A/B/A verified).

Possibly related to the GSP stutter reports in #777 and #693 — the stall signature (multi-ms, whole-system, driver-triggered) looks like a long non-preemptible section around GPU/GSP communication, and nvidia-powerd's frequent polling multiplies the trigger rate.

Environment

  • Laptop: Lenovo Legion Pro 7 16IAX10H (Arrow Lake-S + RTX 5080 Laptop GPU, hybrid graphics), BIOS Q7CN77WW (2026-02-11)
  • GPU: NVIDIA GeForce RTX 5080 Laptop GPU
  • Driver: 610.43.03, open kernel module (NVRM: NVIDIA UNIX Open Kernel Module for x86_64 610.43.03)
  • Kernel: 7.1.3-2-cachyos (Arch-based CachyOS), PipeWire 1.6.8, KDE Plasma / Wayland
  • Workload while reproducing: Palworld under Proton, music playback in Firefox

How the stalls were measured

RT-priority latency canary: a SCHED_FIFO 70 thread doing a 1 ms clock_nanosleep loop, logging every wakeup that overshoots by >3 ms:

import time, sys
with open(sys.argv[1], "a", buffering=1) as f:
    while True:
        t0 = time.monotonic_ns()
        time.sleep(0.001)
        dt = time.monotonic_ns() - t0 - 1_000_000
        if dt > 3_000_000:
            f.write(f"{time.time():.3f} stall={dt/1e6:.1f}ms\n")

Run as taskset -c <cpu> chrt -f 70 python3 canary.py <log>.

Four canaries pinned to CPUs 2, 8, 16, 22 over 7 minutes of gameplay logged 787 / 786 / 787 / 787 stalls respectively, with matching timestamps to within ~1 ms — i.e., every stall freezes all cores at the same instant, uniformly 3.5–6.5 ms long (median 4.2 ms).

Ruled out:

  • SMIs: MSR_SMI_COUNT (0x34) did not increment during a 10 s window in which the canary logged ~20 stalls.
  • Thermal throttling: /sys/devices/system/cpu/cpu0/thermal_throttle/* counters static during stalls; no TRM interrupts.
  • ACPI/EC activity: GPE rate ~0.4/s, uncorrelated.
  • Scheduling/PipeWire config: PipeWire RT threads at RR priority, sink driver reports zero xruns; the xruns land on client streams (all clients simultaneously at stall instants, per pw-top ERR counters).

A/B/A result (same game session, continuous canary)

Condition Stall rate
nvidia-powerd running (baseline, 7 min) ~1.9/s (100–300/min)
systemctl stop nvidia-powerd (11 min) ~0.4–0.8/s (20–50/min)
systemctl start nvidia-powerd (2.5 min) ~3.1/s (460 stalls)

Stall duration distribution is identical in all three conditions (median ~4 ms) — only the frequency changes. Audio dropouts track the stall rate and were subjectively absent with the daemon stopped.

A residual ~0.5 stalls/s remains with nvidia-powerd stopped whenever the dGPU is under load, so the daemon appears to be the highest-frequency trigger of an underlying driver/GSP stall path rather than the sole cause.

Expected behavior

Polling GPU power state (or rebalancing Dynamic Boost budget) should not freeze every CPU core for multiple milliseconds at a time.

Workaround

systemctl disable --now nvidia-powerd — makes audio usable during gaming at the cost of Dynamic Boost.

Happy to provide nvidia-bug-report.sh output, canary logs, or run instrumented builds/driver settings on request.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported stalls with the pinned-CPU canary while running a GPU workload, then compare continuous runs with nvidia-powerd enabled and stopped. Review the referenced GSP reports (#777 and #693) and collect the requested nvidia-bug-report.sh output or instrumented traces. Done means identifying the driver path responsible for the synchronized stalls and preventing the resulting audio dropouts without relying on the workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
operating-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.