NVIDIA / NVIDIA/open-gpu-kernel-modules

GSP: PFM_REQ_HNDLR_STATE_SYNC_CALLBACK missing from _kgspProcessRpcEvent bootup whitelist

Open
#1,145 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
17.4k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

Summary

NV_VGPU_MSG_EVENT_PFM_REQ_HNDLR_STATE_SYNC_CALLBACK (0x101a) is not on the bootup-poll whitelist in _kgspProcessRpcEvent (src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c, around line 1436). When this RPC arrives while CPU-RM is polling for GSP_INIT_DONE without holding the API lock, it falls through to the default: arm, emits a misleading "during bootup without API lock" error, hits NV_ASSERT(0) at kernel_gsp.c:1447, and the RPC is silently dropped, so the Platform Request Handler never sees that wake's state sync.

This is a log-noise / dropped-RPC report, not a crash. The driver continues to function, but the kernel log accumulates two NVRM error lines per affected wake.

Symptom

On a Blackwell laptop dGPU (RTX PRO 1000, driver 595.71.05, NVreg_DynamicPowerManagement=3), runtime-D3 wake-ups produce the following pair:

NVRM: _kgspProcessRpcEvent: Attempted to process RPC event from GPU0: 0x101a (PFM_REQ_HNDLR_STATE_SYNC_CALLBACK) during bootup without API lock
NVRM: nvAssertFailedNoLog: Assertion failed: 0 @ src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c:1447

Roughly two-thirds of runtime-D3 wakes produce the pair; the rest are clean. Timing-dependent, with clusters of all-assert and all-clean consecutive wakes, consistent with the RPC being driven by the platform handler's own sync schedule rather than the wake itself. The "during bootup without API lock" wording is misleading: this is well past initial GSP boot, on a steady-state runtime-D3 wake.

Root cause

The whitelist immediately above the default: arm enumerates the RPCs that are explicitly safe (or expected) to arrive during the bootup poll window. PFM_REQ_HNDLR_STATE_SYNC_CALLBACK is simply absent from that list:

case NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT:
case NV_VGPU_MSG_EVENT_GSP_LOCKDOWN_NOTICE:
case NV_VGPU_MSG_EVENT_GSP_POST_NOCAT_RECORD:
case NV_VGPU_MSG_EVENT_GSP_INIT_DONE:
case NV_VGPU_MSG_EVENT_OS_ERROR_LOG:
case NV_VGPU_MSG_EVENT_GSP_LOAD_EXEC_GENERIC_BOOTLOADER:
case NV_VGPU_MSG_EVENT_GSP_LOAD_EXEC_HS_BINARY:
    break;
default:
    NV_PRINTF(LEVEL_ERROR, "Attempted to process RPC event from GPU%d: 0x%x (%s) during bootup without API lock\n",
              ...);
    NV_ASSERT(0);

Why this handler is safe to run in the bootup-poll context

_kgspRpcEventPlatformRequestHandlerStateSyncCallback decodes the RPC payload and dispatches via pfmreqhndlrStateSync_IMPL on the flags field:

  • For PMGR_LOAD and THERM_INIT, the prereq callbacks only call osQueueWorkItem and return; they touch no shared state inline and make no GSP RPC. The in-tree comment at those callbacks ("we need this to succeed so that we don't have a recursive GSP RPC") guarantees the no-RPC property explicitly.
  • For SMBPBI_OP_SET / SMBPBI_OP_CLEAR, the synchronous portion writes Platform Request Handler counter fields. The polling thread holds the GPU subdevice lock for the duration of the bootup poll, so no concurrent writer to that state can race against the dispatch.

In short: no recursive GSP RPCs, no unprotected shared-state writes. The handler is well-behaved under the constraints the bootup-poll whitelist is enforcing.

Suggested fix

diff --git a/src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c b/src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c
index ab1c3695..b3c36405 100644
--- a/src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c
+++ b/src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c
@@ -1440,6 +1440,7 @@ _kgspProcessRpcEvent
             case NV_VGPU_MSG_EVENT_OS_ERROR_LOG:
             case NV_VGPU_MSG_EVENT_GSP_LOAD_EXEC_GENERIC_BOOTLOADER:
             case NV_VGPU_MSG_EVENT_GSP_LOAD_EXEC_HS_BINARY:
+            case NV_VGPU_MSG_EVENT_PFM_REQ_HNDLR_STATE_SYNC_CALLBACK:
                 break;
             default:
                 NV_PRINTF(LEVEL_ERROR, "Attempted to process RPC event from GPU%d: 0x%x (%s) during bootup without API lock\n",

Verification

Built 595.71.05 with the diff applied, ~3 hours of normal use including many runtime-D3 wake cycles. Zero kernel_gsp.c:1447 / 0x101a lines after the patch, vs roughly two-thirds of wakes producing the pair before. No regressions observed.

Relation to issue #1064

This fix is not a fix for #1064. #1064 is a DIFR pmlock deadlock that wedges Mutter on nvkms_lock and requires SysRq REISUB to recover; the deadlock reproduces with this patch applied. The two are unrelated: the only connection is that #1064 was the path by which the missing whitelist entry got noticed. Tracking that separately on the #1064 thread.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c around _kgspProcessRpcEvent at line 1436 and inspect the bootup-poll whitelist and the PFM state-sync callback handler. Build the driver with the callback included in the whitelist, then exercise runtime-D3 wake cycles and confirm the 0x101a and kernel_gsp.c:1447 messages no longer appear.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.