Laptop freezes after upgrade to 525.60.11

NVIDIA Open GPU Kernel Modules Version

525.60.11

Does this happen with the proprietary driver (of the same version) as well?

I cannot test this

Operating System and Version

Arch Linux

Kernel Release

6.0.11-arch1-1

Hardware: GPU

3060 Ti

Describe the bug

I upgraded to 525,60.11 a couple of weeks and my laptop (Lenovo Slim 7) with an eGPU using a 3060Ti freezes within a few minutes of booting, When I reboot after I see the following in the previous boot logs:

Dec 08 13:00:44 gnunn-lenovo kernel: NVRM: GPU at PCI:0000:2d:00: GPU-275eb527-5847-0b7e-27a3-fb54afd72bac
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM: GPU Board Serial Number: 1322321046802
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM: Xid (PCI:0000:2d:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM: GPU 0000:2d:00.0: GPU has fallen off the bus.
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM: GPU 0000:2d:00.0: GPU serial number is 1322321046802.
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: !pKernelGsp->bPollingForRpcResponse @ kernel_gsp.c:1158
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM _issueRpcAndWait: rpcRecvPoll failed with status 0x00000040 for fn 78!
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: 0 @ rpc.c:204
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvCheckOkFailedNoLog: Check failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:272
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: !pKernelGsp->bPollingForRpcResponse @ kernel_gsp.c:1158
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM _issueRpcAndWait: rpcRecvPoll failed with status 0x00000040 for fn 78!
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: 0 @ rpc.c:204
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvCheckOkFailedNoLog: Check failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:272
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: !pKernelGsp->bPollingForRpcResponse @ kernel_gsp.c:1158
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM _issueRpcAndWait: rpcRecvPoll failed with status 0x00000040 for fn 78!
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: 0 @ rpc.c:204
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM nvCheckOkFailedNoLog: Check failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:272
Dec 08 13:00:44 gnunn-lenovo kernel: NVRM RmLogGpuCrash: RmLogGpuCrash: failed to save GPU crash data
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: expectedFunc == pRpc->rpcHistory[historyEntry].function @ kernel_gsp.c:1077
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM: Xid (PCI:0000:2d:00): 119, pid=2562, name=gnome-shell, Timeout waiting for RPC from GSP! Expected function 76 (GSP_RM_CONTROL) (0x0 0x0).
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119: Current RPC function 78 (DUMP_PROTOBUF_COMPONENT) or data (0x0 0x0) does not match expected function 76 (GSP_RM_CONTROL) or data (0x0 0x0).
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119: RPC history (CPU -> GSP):
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         entry        func                                data
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:          0         78 DUMP_PROTOBUF_COMPONENT        0x00000000 0x00000000
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         -1         78 DUMP_PROTOBUF_COMPONENT        0x00000000 0x00000000
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         -2         78 DUMP_PROTOBUF_COMPONENT        0x00000000 0x00000000
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         -3         76 GSP_RM_CONTROL                0x00730109 0x0000000c
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         -4         76 GSP_RM_CONTROL                0x00730109 0x0000000c
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         -5         76 GSP_RM_CONTROL                0x00730109 0x0000000c
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         -6         76 GSP_RM_CONTROL                0x00730109 0x0000000c
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119:         -7         76 GSP_RM_CONTROL                0x00730109 0x0000000c
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _kgspLogXid119: Dumping stack:
Dec 08 13:00:50 gnunn-lenovo kernel: CPU: 4 PID: 2562 Comm: gnome-shell Tainted: G     U     OE      6.0.11-arch1-1 #1 7fb03aecebbbd5f8fb3d28f15ac14202b3cf258e
Dec 08 13:00:50 gnunn-lenovo kernel: Hardware name: LENOVO 82SX/LNVNB161216, BIOS JHCN24WW 07/18/2022
Dec 08 13:00:50 gnunn-lenovo kernel: Call Trace:
Dec 08 13:00:50 gnunn-lenovo kernel:  <TASK>
Dec 08 13:00:50 gnunn-lenovo kernel:  dump_stack_lvl+0x48/0x60
Dec 08 13:00:50 gnunn-lenovo kernel:  _kgspRpcRecvPoll+0x49e/0x5e0 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  _issueRpcAndWait+0x3a/0x1e0 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  rpcRmApiControl_GSP+0x24e/0x7e0 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  rmresControl_Prologue_IMPL+0x109/0x250 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  resControl_IMPL+0xd2/0x1b0 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  dispapiControl_IMPL+0x12d/0x170 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  serverControl+0x264/0x390 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  _rmapiRmControl+0x6db/0x7e0 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  rmapiControlWithSecInfo+0x7a/0x140 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  rmapiControlWithSecInfoTls+0x85/0x110 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  _nv04ControlWithSecInfo+0xb2/0xd0 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  RmIoctl+0x5c4/0xd60 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  ? osGetCurrentTick+0x15/0x20 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  ? os_acquire_spinlock+0x12/0x30 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  rm_ioctl+0x58/0x90 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  nvidia_ioctl+0x6a9/0x870 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  nvidia_frontend_unlocked_ioctl+0x39/0x50 [nvidia 95630f1563c0edef3472b06d559144f13f7cc25d]
Dec 08 13:00:50 gnunn-lenovo kernel:  __x64_sys_ioctl+0x91/0xd0
Dec 08 13:00:50 gnunn-lenovo kernel:  do_syscall_64+0x5c/0x90
Dec 08 13:00:50 gnunn-lenovo kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
Dec 08 13:00:50 gnunn-lenovo kernel:  ? do_syscall_64+0x6b/0x90
Dec 08 13:00:50 gnunn-lenovo kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
Dec 08 13:00:50 gnunn-lenovo kernel:  ? do_syscall_64+0x6b/0x90
Dec 08 13:00:50 gnunn-lenovo kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
Dec 08 13:00:50 gnunn-lenovo kernel:  ? do_syscall_64+0x6b/0x90
Dec 08 13:00:50 gnunn-lenovo kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
Dec 08 13:00:50 gnunn-lenovo kernel:  ? do_syscall_64+0x6b/0x90
Dec 08 13:00:50 gnunn-lenovo kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
Dec 08 13:00:50 gnunn-lenovo kernel:  ? do_syscall_64+0x6b/0x90
Dec 08 13:00:50 gnunn-lenovo kernel:  ? __irq_exit_rcu+0x4b/0xe0
Dec 08 13:00:50 gnunn-lenovo kernel:  entry_SYSCALL_64_after_hwframe+0x63/0xcd
Dec 08 13:00:50 gnunn-lenovo kernel: RIP: 0033:0x7f277e115c0f
Dec 08 13:00:50 gnunn-lenovo kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00
Dec 08 13:00:50 gnunn-lenovo kernel: RSP: 002b:00007ffdd3d8c660 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Dec 08 13:00:50 gnunn-lenovo kernel: RAX: ffffffffffffffda RBX: 0000000000000020 RCX: 00007f277e115c0f
Dec 08 13:00:50 gnunn-lenovo kernel: RDX: 00007ffdd3d8c7b0 RSI: 00000000c020462a RDI: 000000000000000d
Dec 08 13:00:50 gnunn-lenovo kernel: RBP: 00007ffdd3d8c7b0 R08: 00007ffdd3d8c7b0 R09: 00007ffdd3d8c7cc
Dec 08 13:00:50 gnunn-lenovo kernel: R10: 00007f275ee6bfc0 R11: 0000000000000246 R12: 000000000000000d
Dec 08 13:00:50 gnunn-lenovo kernel: R13: 00000000c020462a R14: 000000000000002a R15: 00007ffdd3d8c6c0
Dec 08 13:00:50 gnunn-lenovo kernel:  </TASK>
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM _issueRpcAndWait: rpcRecvPoll timedout for fn 78!
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: 0 @ rpc.c:204
Dec 08 13:00:50 gnunn-lenovo kernel: NVRM nvAssertFailedNoLog: Assertion failed: status == NV_OK @ video_mem.c:141

To Reproduce

Install the updated driver, boot into X and start using the system.

Bug Incidence

Always

nvidia-bug-report.log.gz

Since system is frozen I cannot do this

More Info

Reverting to 520 everything works fine.