loading /lib/firmware/nvidia/570.133.20/gsp_ga10x.bin failed with error -4

NVIDIA Open GPU Kernel Modules Version

570.172.08

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

No LSB modules are available. Description: Ubuntu 24.04.2 LTS

Kernel Release

Linux cvm 6.8.0-85-generic #85-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 18 15:26:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

nvidia-smi -L GPU 0: NVIDIA H200 (UUID: GPU-e9c8dd70-5424-0717-37e2-0a2b101e152f) GPU 1: NVIDIA H200 (UUID: GPU-c4690ddf-5490-8dfb-6527-a70ff5248219) GPU 2: NVIDIA H200 (UUID: GPU-737e21a0-ace6-ffec-2f2b-b85d2bfc7649) GPU 3: NVIDIA H200 (UUID: GPU-6bc4a1f0-256e-4106-3725-0c2942705741) GPU 4: NVIDIA H200 (UUID: GPU-ad2b7891-04a5-7899-f311-24a1410e62ab)

It should shows 8 gpus, but turns out 5 shown only

Describe the bug

[   81.335676] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  570.172.08  Release Build  (dvs-builder@U22-I3-AF01-21-3)  Tue Jul  8 17:59:47 UTC 2025
[   81.379549] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[   81.915785] ACPI Warning: \_SB.PCI0.S18.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[  110.628102] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  110.628131] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  112.430325] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 0
[  112.430354] nvidia 0000:01:00.0: [drm] No compatible format found
[  112.430359] nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes
[  112.431143] [drm] [nvidia-drm] [GPU ID 0x00000200] Loading driver
[  112.827594] ACPI Warning: \_SB.PCI0.S19.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[  140.713584] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  140.713613] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  142.957348] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:02:00.0 on minor 1
[  142.957378] nvidia 0000:02:00.0: [drm] No compatible format found
[  142.957383] nvidia 0000:02:00.0: [drm] Cannot find any crtc or sizes
[  142.958190] [drm] [nvidia-drm] [GPU ID 0x00000300] Loading driver
[  143.351850] ACPI Warning: \_SB.PCI0.S1A.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[  171.491409] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  171.491439] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  174.237193] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:03:00.0 on minor 2
[  174.237218] nvidia 0000:03:00.0: [drm] No compatible format found
[  174.237222] nvidia 0000:03:00.0: [drm] Cannot find any crtc or sizes
[  174.238645] [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver
[  174.634643] ACPI Warning: \_SB.PCI0.S1B.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[  202.909622] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  202.909657] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  206.198059] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:04:00.0 on minor 3
[  206.198096] nvidia 0000:04:00.0: [drm] No compatible format found
[  206.198100] nvidia 0000:04:00.0: [drm] Cannot find any crtc or sizes
[  206.198803] [drm] [nvidia-drm] [GPU ID 0x00000500] Loading driver
[  206.587998] ACPI Warning: \_SB.PCI0.S1C.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[  232.165176] systemd-journald[2045]: /var/log/journal/502ddb25b848499f9617d76c3b27b682/user-1000.journal: Journal file uses a different sequence number ID, rotating.
[  234.780660] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  234.780686] NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_gmmu_nvoc.h:2256
[  238.662777] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:05:00.0 on minor 4
[  238.662812] nvidia 0000:05:00.0: [drm] No compatible format found
[  238.662819] nvidia 0000:05:00.0: [drm] Cannot find any crtc or sizes
[  238.664064] [drm] [nvidia-drm] [GPU ID 0x00000600] Loading driver
[  238.717429] nvidia 0000:06:00.0: loading /lib/firmware/updates/nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.742589] nvidia 0000:06:00.0: loading /lib/firmware/nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.742605] nvidia 0000:06:00.0: Direct firmware load for nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.742623] NVRM: RmFetchGspRmImages: No firmware image found
[  238.742765] NVRM: GPU 0000:06:00.0: RmInitAdapter failed! (0x61:0x56:1770)
[  238.750716] NVRM: GPU 0000:06:00.0: rm_init_adapter failed, device minor number 5
[  238.759003] [drm:nv_drm_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000600] Failed to allocate NvKmsKapiDevice
[  238.766228] [drm:nv_drm_register_drm_device [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000600] Failed to register device
[  238.775988] [drm] [nvidia-drm] [GPU ID 0x00000700] Loading driver
[  238.827032] nvidia 0000:07:00.0: loading /lib/firmware/updates/nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.849396] nvidia 0000:07:00.0: loading /lib/firmware/nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.849414] nvidia 0000:07:00.0: Direct firmware load for nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.849435] NVRM: RmFetchGspRmImages: No firmware image found
[  238.849562] NVRM: GPU 0000:07:00.0: RmInitAdapter failed! (0x61:0x56:1770)
[  238.859194] NVRM: GPU 0000:07:00.0: rm_init_adapter failed, device minor number 6
[  238.868979] [drm:nv_drm_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000700] Failed to allocate NvKmsKapiDevice
[  238.877240] [drm:nv_drm_register_drm_device [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000700] Failed to register device
[  238.885251] [drm] [nvidia-drm] [GPU ID 0x00000800] Loading driver
[  238.944361] nvidia 0000:08:00.0: loading /lib/firmware/updates/nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.963752] nvidia 0000:08:00.0: loading /lib/firmware/nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.963760] nvidia 0000:08:00.0: Direct firmware load for nvidia/570.172.08/gsp_ga10x.bin failed with error -4
[  238.963893] NVRM: RmFetchGspRmImages: No firmware image found
[  238.963982] NVRM: GPU 0000:08:00.0: RmInitAdapter failed! (0x61:0x56:1770)
[  238.971855] NVRM: GPU 0000:08:00.0: rm_init_adapter failed, device minor number 7
[  238.979302] [drm:nv_drm_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000800] Failed to allocate NvKmsKapiDevice
[  238.986998] [drm:nv_drm_register_drm_device [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000800] Failed to register device

To Reproduce

everytime it boot, this happened

Bug Incidence

Once

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

The payload names no source files, tests, or entry points. Start by examining the reported firmware-load failures for PCI devices 0000:06:00.0 through 0000:08:00.0 and the attached nvidia-bug-report.log.gz; done means all eight NVIDIA H200 GPUs initialize without the gsp_ga10x.bin errors.