Nvidia DMA is not working with AMD memory encryption (SME)

NVIDIA Open GPU Kernel Modules Version

515.57

Does this happen with the proprietary driver (of the same version) as well?

Yes

Operating System and Version

Arch Linux

Kernel Release

5.17.9-arch1-1

Hardware: GPU

NVIDIA GeForce GTX 1660 SUPER

Describe the bug

I have a Ryzen 3660, which supports SME. I've tried activating it with the mem_encrypt=on kernel boot option. The system loads fine and dmesg reports that SME is active:

[    0.122077] AMD Memory Encryption Features active: SME

I also have IOMMU enabled in the UEFI, and dmesg reports that it's working. However, if I try to start the Xorg server, it fails with the following error:

[    21.357] (II) NVIDIA: Reserving 24576.00 MB of virtual memory for indirect memory
[    21.357] (II) NVIDIA:     access.
[    24.364] (EE) NVIDIA(GPU-0): Failed to initialize DMA.
[    24.365] (EE) NVIDIA(0): Failed to allocate push buffer

I've tried this with Linux version 5.17.9-arch1-1 or 5.18.14-arch1-1, and with Nvidia 515.57. For Nvidia I've tried both closed sourced and open sources drivers, as well as non-dkms and dkms versions. I've also tried recompiling the open source drivers. I always get the same error.

I've found online this earlier post on the gentoo bugzilla: https://bugs.gentoo.org/652408. As far as I read from that post, the driver should work with memory encryption enabled. I've also glanced at the source code, and there is a small code section with #ifdef on CONFIG_AMD_MEM_ENCRYPT, which suggests the code at least intends to work with AMD SME. That code also checks NV_PGPROT_DECRYPTED_PRESENT, which seems undefined by default. I've tried defining it and recompiling the code, but I still get the same error.

To Reproduce

Enable SME on a Ryzon processor using the mem_encrypt=on kernel boot option, start the Xorg sever.

Bug Incidence

Always

nvidia-bug-report.log.gz

(Not including the output, because it collects everything about my computer and I do not feel comfortable providing all that identifying information. I'm happy to provide a specific part, if required. Though I do not expect the issue to be specific to my computer, seems like a general issue of whether the recent driver works with the recent kernel when AMD SME is enabled.)

More Info

No response

Start by reproducing the failure with mem_encrypt=on, IOMMU enabled, Xorg, and the listed kernel and NVIDIA driver versions, then inspect the dmesg DMA errors. Read the driver source around CONFIG_AMD_MEM_ENCRYPT and NV_PGPROT_DECRYPTED_PRESENT, comparing the open and proprietary-driver behavior described here. Done means identifying and fixing the SME-related DMA initialization failure, with Xorg starting successfully.