HMM causes CUDA initialization failures on EL9 systems

NVIDIA Open GPU Kernel Modules Version

570.158.01, 575.57.08

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Rocky 9.6

Kernel Release

5.14.0-570.23.1.el9_6.x86_64

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

RTX 6000 Ada Gen, RTX 4500 Ada Gen

Describe the bug

All CUDA calls fail on stable EL9 systems with the latest versions of both the nvidia-open and proprietary drivers. Drivers were installed via the documentation, and the issue exists with both the cuda-rhel9-x86_64 packages and runtime installers. nvidia-smi and /dev directories both show the cards are detected and the drivers load without issue. journald, /var/log/message, and dmesg indicate no issues with the driver kernel modules at boot or during runtime. However, trying to use CUDA workflows with PyTorch or cuda-toolkit will fail and report that no CUDA devices could be found. Disabling HMM via the kernel module option nvidia_uvm uvm_disable_hmm=1 and reloading the nvidia_uvm module fixes this issue and allows CUDA calls to work.

We encountered this issue on two separate systems in our datacenter and with different types of GPUs.

To Reproduce

Install the nvidia-open drivers and cuda-toolkit on 5.14.0 EL9 system using any method available in the documentation. Clone the NVIDIA/cuda-samples repo, compile the binaries, and run the deviceQuery binary. Running with a debugger (gdb) will provide additional information on the error as well as traces. You can confirm functionality without HMM by disabling this feature with the nvidia_uvm uvm_disable_hmm=1 kernel module option.

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

Start by reproducing the failure on the reported Rocky 9.6 kernel with the NVIDIA drivers and cuda-toolkit, then run deviceQuery from NVIDIA/cuda-samples with HMM enabled and with nvidia_uvm uvm_disable_hmm=1. Inspect the nvidia_uvm behavior and the supplied nvidia-bug-report logs, journald, and dmesg. Done means CUDA detects and uses the GPUs with HMM enabled on the affected EL9 systems.