VFIO GPU reads invalid ROM signature

NVIDIA Open GPU Kernel Modules Version

575.64.03

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 22.04.5 LTS

Kernel Release

6.8.0-79

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

RTX 3090

Describe the bug

Hi, NV's developers,
I encountered an issue while testing GPU VFIO passthrough to a VM.
I'm using Kata-QEMU to create the VM, and Kata adds the GPU to the VM via hot-plug.
The GPU is an RTX 3090, and I passed through both the GPU and the Audio Controller to the VM.
Sometimes the GPU driver reports an error: kgspExtractVbiosFromRom_TU102: did not find valid ROM signature.
This behavior occurs intermittently—about three times out of ten boot attempts.
Details are shown in the log below.

[   19.287730] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  575.64.03  Release Build  (dvs-builder@U22-I3-H04-03-5)  Wed Jun 25 18:07:44 UTC 2025
[   19.288229] initcall nvkms_init+0x0/0x1000 [nvidia_modeset] returned 0 after 709 usecs
[   19.594210] NVRM: kgspExtractVbiosFromRom_TU102: did not find valid ROM signature
[   19.594499] NVRM: kgspInitRm_IMPL: failed to extract VBIOS image from ROM: 0x25
[   19.594718] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[   19.595209] NVRM: GPU 0000:02:00.0: RmInitAdapter failed! (0x62:0x25:1941)
[   19.596014] NVRM: GPU 0000:02:00.0: rm_init_adapter failed, device minor number 0
[   19.620355] NVRM: kgspExtractVbiosFromRom_TU102: did not find valid ROM signature
[   19.620606] NVRM: kgspInitRm_IMPL: failed to extract VBIOS image from ROM: 0x25
[   19.620811] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[   19.621243] NVRM: GPU 0000:02:00.0: RmInitAdapter failed! (0x62:0x25:1941)
[  19.622090] NVRM: GPU 0000:02:00.0: rm_init_adapter failed, device minor number 0
[   19.641691] NVRM: kgspExtractVbiosFromRom_TU102: did not find valid ROM signature
[   19.641951] NVRM: kgspInitRm_IMPL: failed to extract VBIOS image from ROM: 0x25
[   19.642156] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[   19.642702] NVRM: GPU 0000:02:00.0: RmInitAdapter failed! (0x62:0x25:1941)
[   19.643856] NVRM: GPU 0000:02:00.0: rm_init_adapter failed, device minor number 0
[   19.652541] NVRC::lockdown[162]: panic: panicked at src/main.rs:70:9:
[   19.652541] init failure: nvidia_ctk_cdi() => /bin/nvidia-ctk failed with status: exit status: 1

Does anyone know why the driver is reading incorrect vBIOS?

To Reproduce

1. Install the kata and containerd

ctr --debug run --runtime "io.containerd.kata.v2" \
    --device /dev/vfio/41 --rm -t \
    arch nvidia-smi

Bug Incidence

Sometimes

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

Start with the Kata-QEMU reproduction command using /dev/vfio/41 and compare successful and failing boot attempts. Review the attached nvidia-bug-report.log.gz alongside the repeated kgspExtractVbiosFromRom_TU102 errors; done means the intermittently hot-plugged RTX 3090 initializes without the invalid ROM signature failure.