Non-prefetchable memory in 3rd memory mapped bar results in loop failure

NVIDIA Open GPU Kernel Modules Version

latest.

Does this happen with the proprietary driver (of the same version) as well?

Yes

Operating System and Version

Ubuntu 20.04

Kernel Release

5.15

Hardware: GPU

A100-40gb on SXM4

Describe the bug

The NVIDIA driver expects the final bar to be marked as prefetchable. In some environments, such as virtualized environments, it may make sense to mark a bar as prefetchable. The PCI Express Specifation 4.0 R1.0 Septermber 2017 page 704 indicates NVIDIA hardware should be marking these BARs as prefetchable. Unfortunately, the driver logic does not tolerate a situation where they are not prefetchable.

See: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/4451 for an analysis of the code and. the problem.

To Reproduce

Launch cloud-hypervisor, which can be found at https://github.com/cloud-hypervisor/cloud-hypervisor with VFIO passthrough of a nvidia device. The driver will display:

[    1.112630] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    1.119271] nvidia-nvlink: Nvlink Core is being initialized, major device number 240
[    1.121034] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
               NVRM: BAR2 is 0M @ 0x0 (PCI:0000:00:05.0)
[    1.122691] nvidia: probe of 0000:00:05.0 failed with error -1
[    1.123500] NVRM: The NVIDIA probe routine failed for 1 device(s).
[    1.124330] NVRM: None of the NVIDIA devices were initialized.

The above loops flooding out the console.

This problem can be resolved by applying a patch to cloud-hypervisor to force VFIO devices to prefetchable which is likely the correct state for the BAR. However if the bar is not prefetchable, the above occurs, which is not ideal.

sudo ./cloud-hypervisor-main --kernel ./hypervisor-fw \
	--disk path=./lambdastack.raw \
        --cmdline "module_blacklist=nouveau console=hvc0 root=/dev/vda1 rw" \
	--cpus "boot=16" \
	--memory "size=500G,hugepages=on,shared=on" \
	--net "tap=,mac=,ip=,mask=" \
	-v \
	--log-file "/home/ubuntu/chv.log" \
	--device "path=/sys/bus/pci/devices/0000:15:00.0/"

where --device is the PCI device that is being passed through, and bound to VFIO correctly as per https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/vfio.md

Bug Incidence

Always

nvidia-bug-report.log.gz

I do not have this tool.

More Info

No response

Reproduce with cloud-hypervisor using the supplied VFIO passthrough command on Ubuntu 20.04 and kernel 5.15, then inspect the NVIDIA probe path around the BAR2 validation reported in the NVRM log. Done means the non-prefetchable BAR case no longer causes the shown probe failure and repeated console output.