RmInitAdapter failed!

NVIDIA Open GPU Kernel Modules Version

NVIDIA-Linux-x86_64-550.144.03

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 22.04

Kernel Release

5.15.0-131-generic

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

3090

Describe the bug

invidia-smi , No devices were found, RmInitAdapter failed!

To Reproduce

System: Ubuntu22.04
GPU: 3090
NVIDIA DRIVER VERSION: 550.114.03
KERNEL VERSION: 5.15.0.131

nouveau is also disabled and safe mode is turned off
When I input invidia-smi, an error message says No devices were found
dmesg output error:
[ 903.609564] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[ 903.610907] NVRM: GPU 0000:98:00.0: RmInitAdapter failed! (0x62:0x65:1792)
[ 903.611884] NVRM: GPU 0000:98:00.0: rm_init_adapter failed, device minor number 0
[ 2052.745833] NVRM: gpuWaitForGfwBootComplete_TU102: failed to wait for GFW_BOOT: (progress 0x3)
[ 2052.745838] NVRM: kgspWaitForGfwBootOk_TU102: failed to wait for GFW boot complete: 0x65 VBIOS version 94.02.59.00.05
[ 2052.745839] NVRM: kgspWaitForGfwBootOk_TU102: (the GPU may be in a bad state and may need to be reset)
[ 2052.745842] NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgspWaitForGfw

install log also reports an error when installing with the NVIDIA-Linux-x86_64-550.144.03.run program
[ 320.517169] NVRM: GPU 0000:98:00.0: RmInitAdapter failed! (0x62:0x65:1792)
[ 320.518037] NVRM: GPU 0000:98:00.0: rm_init_adapter failed, device minor number 0
[ 811.013645] NVRM: gpuWaitForGfwBootComplete_TU102: failed to wait for GFW_BOOT: (progress 0x3)
[ 811.013650] NVRM: kgspWaitForGfwBootOk_TU102: failed to wait for GFW boot complete: 0x65 VBIOS version 94.02.59.00.05
[ 811.013651] NVRM: kgspWaitForGfwBootOk_TU102: (the GPU may be in a bad state and may need to be reset)
[ 811.013653] NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgspWaitForGfwBootOk_HAL(pGpu, pKernelGsp) @ kernel_gsp.c:3199
[ 811.013657] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[ 811.014976] NVRM: GPU 0000:98:00.0: RmInitAdapter failed! (0x62:0x65:1792)
[ 811.015905] NVRM: GPU 0000:98:00.0: rm_init_adapter failed, device minor number 0
[ 911.253427] nvidia-uvm: Unloaded the UVM driver.
[ 911.341744] [drm] [nvidia-drm] [GPU ID 0x00009800] Unloading driver
[ 911.389814] nvidia-modeset: Unloading
[ 911.487798] nvidia-nvlink: Unregistered Nvlink Core, major device number 507
[ 1157.177118] nvidia-nvlink: Nvlink Core is being initialized, major device number 507
[ 1157.183881] nvidia 0000:98:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none
[ 1158.697433] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 550.144.03 Release Build (dvs-builder@U16-I3-D08-1-2) Mon Dec 30 17:26:13 UTC 2024
[ 1158.734861] nvidia-uvm: Loaded the UVM driver, major device number 505.
[ 1158.749223] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 550.144.03 Release Build (dvs-builder@U16-I3-D08-1-2) Mon Dec 30 17:11:39 UTC 2024
[ 1158.753539] [drm] [nvidia-drm] [GPU ID 0x00009800] Loading driver
[ 1158.753543] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:98:00.0 on minor 0
[ 1158.757865] [drm] [nvidia-drm] [GPU ID 0x00009800] Unloading driver

Please help me see the cause of the problem，Thanks

Bug Incidence

Always

nvidia-bug-report.log.gz

dmesg.log
nvidia-bug-report.log.gz
nvidia-installer.log

More Info

No response

Start by reviewing the NVIDIA kernel driver's GSP initialization path, including kernel_gsp.c around line 3199, and inspect the attached dmesg.log, nvidia-bug-report.log.gz, and nvidia-installer.log. Reproduce with Ubuntu 22.04, kernel 5.15.0-131-generic, GPU 3090, and driver 550.144.03; done means identifying and fixing or documenting the cause of the GSP firmware timeout so invidia-smi detects the device.