dma-buf export returns unmapped BAR addresses to the importer under IOMMU translation (595.84 open, kernel 7.0)

NVIDIA Open GPU Kernel Modules Version

595.84

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 26.04.1 LTS

Kernel Release

Linux 7.0.0-30-generic #30-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 31 18:22:54 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA RTX 2000 Ada Generation (UUID: GPU-54cdb24f-81b6-9400-2e09-78e819aeca9a)

Describe the bug

dma-buf export returns untranslated BAR addresses to the importer (595.84 open, kernel 7.0)

Summary

A PCIe device importing a CUDA allocation via cuMemGetHandleForAddressRange(..., CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0) receives, in sg_dma_address(), raw physical
addresses inside the GPU's BAR1 instead of addresses mapped for the importing device.
With the IOMMU translating, every peer write is then discarded by VT-d.

The returned addresses are byte-for-byte identical whether the IOMMU translates or not,
which is the shortest statement of the problem: nothing is adjusted for the importer's
IOMMU domain.

iommu=pt masks it, because there the physical address happens to be the right one.

Environment

Kernel	7.0.0-30-generic (Ubuntu)
NVIDIA	595.84, open kernel modules (nvidia-595-open)
GPU	RTX 2000 Ada (AD107GL) at 0000:01:00.0, BAR1 0x40000000-0x4fffffff
Peer	Third-party PCIe device at 0000:02:00.0, bus master, out-of-tree driver
Topology	Separate root ports (00:01.0, 00:01.1), single IOMMU group
IOMMU	Intel VT-d, CONFIG_INTEL_IOMMU_DEFAULT_ON=y, domain type DMA-FQ

Reproduce

User space, flags 0 (no CU_MEM_RANGE_FLAG_DMA_BUF_MAPPING_TYPE_PCIE):

cuMemGetHandleForAddressRange(&Fd, DevPtr, Size, CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0);

Importer, the ordinary sequence:

Buf     = dma_buf_get(Fd);
Attach  = dma_buf_attach(Buf, &pdev->dev);          /* the bus-mastering device */
SgTable = dma_buf_map_attachment_unlocked(Attach, DMA_FROM_DEVICE);
for_each_sgtable_dma_sg(SgTable, Sg, i)             /* sg_dma_address / sg_dma_len only */

A 64 MB allocation returns 1024 segments of 65536 bytes.

With iommu=pt:

seg 0: dma 0x41720000, 65536 bytes
seg 1: dma 0x41730000, 65536 bytes

Without iommu=pt, same machine, same run:

seg 0: dma 0x41720000, 65536 bytes   iommu_iova_to_phys() = 0
seg 1: dma 0x41730000, 65536 bytes   iommu_iova_to_phys() = 0

Identical addresses, and none resolve in the importing device's domain. 0x41720000 is
inside the GPU's BAR1 window. The device writes there and VT-d drops it:

dmar_fault: 1133498 callbacks suppressed
DMAR: [DMA Write NO_PASID] Request device [02:00.0] fault addr 0x41720000
      [fault reason 0x05] PTE Write access is not set

Every call succeeds. dma_buf_get, dma_buf_attach and dma_buf_map_attachment_unlocked
all return normally. The only symptom is that no data arrives.

Why we believe this is the exporter's mapping to make

Stated because it is the load-bearing assumption, and it was checked before filing:

• Kerneldoc for map_dma_buf: "A sg_table scatter list of the backing storage of the
  DMA buffer, already mapped into the device address space of the device attached with the
  provided dma_buf_attachment."
• The in-tree reference importer, drivers/infiniband/core/umem_dmabuf.c, does not
  re-map: it uses sg_dma_address() / sg_dma_len() from the returned table directly.
• An importer that calls dma_map_* on that table would be double-mapping, and these
  entries have no struct page to map from in any case.

What is ruled out

• FORCE_PCIE / skip_iommu — the export flags argument is 0. Per nv_dma_buf_attach(),
  skip_iommu is set only by NV_DMABUF_EXPORT_MAPPING_TYPE_FORCE_PCIE.
• Plain vs dynamic attach. We use dma_buf_attach(), where umem_dmabuf.c uses
  dma_buf_dynamic_attach() with allow_peer2peer = true. Per nv_dma_buf_attach(), the
  peer2peer test only refuses an attach (-ENOTSUPP) and only when importer_ops != NULL;
  it does not change how addresses are produced. Our attach is accepted.
• Double mapping by us — no dma_map_sg / dma_map_sgtable / dma_map_resource is
  called on the returned table.
• dma_map_resource() being unavailable on this kernel — see below, it works.

The measurement that narrows it

nv_dma_map_peer() calls nv_dma_map_mmio() when nv_dma_use_map_resource() is true, and
otherwise falls through to

/* Best effort - can't map through the iommu but at least try to convert to a bus address. */
NvU64 offset = *va - res->start;
*va = pci_bus_address(peer_pci_dev, bar_index) + offset;

On x86-64 that yields the host physical address unchanged — exactly what we observe. So the
fallback is being taken and nv_dma_use_map_resource() returns NV_FALSE.

But on this system it should return NV_TRUE. From our own driver, on the importing
device:

get_dma_ops() is NULL for this device

and nv_dma_use_map_resource() as published reads:

const struct dma_map_ops *ops = get_dma_ops(dma_dev->dev);
if (nv_dma_remap_peer_mmio == NV_DMA_REMAP_PEER_MMIO_DISABLE) return NV_FALSE;
if (ops == NULL) return NV_TRUE;   /* "we can utilize direct dma" */

With ops == NULL the function returns true before it ever reaches the map_phys /
map_resource test, so the kernel's move to the physical-address DMA API is not the
explanation either.

The remaining guard in that function is nv_dma_remap_peer_mmio. It is not the
explanation. The module exposes no sysfs entry for the parameter
(/sys/module/nvidia/parameters/ does not exist), so rather than read it we set it
explicitly on the kernel command line and rebooted:

$ cat /proc/cmdline
BOOT_IMAGE=... ro nvidia.NVreg_DmaRemapPeerMmio=1 quiet splash ...

Behaviour is unchanged: the addresses still do not resolve, and are still the raw BAR
physical ones.

So with the export flags at 0, NVreg_DmaRemapPeerMmio explicitly 1, and get_dma_ops()
NULL, every branch of nv_dma_use_map_resource() as published returns NV_TRUE — and the
address we get back is the one only its NV_FALSE path can produce.

That leaves one conclusion we cannot check from outside the module: either
nv_dma_use_map_resource() in 595.84 does not behave as main publishes it, or this
configuration does not reach nv_dma_map_peer() at all.

What makes it work, for reference

Mapping each segment from the importing driver:

dma_map_resource(&pdev->dev, sg_dma_address(Sg), sg_dma_len(Sg), DMA_FROM_DEVICE, 0);

0x41720000 becomes a real IOVA, the faults go to zero and the data lands. So
dma_map_resource() works on this kernel for this device — it simply is not being reached
inside the module.

This is a workaround in the wrong layer, and we only consider it safe because the condition
is detectable: we ask iommu_iova_to_phys() whether the first address resolves, and leave
the addresses untouched when it does. A release that maps them will be handled correctly
with no change on our side.

Asks

1. Confirm whether nv_dma_use_map_resource() and nv_dma_map_peer() in 595.84 match what
   is published on main, and if so, which branch produces an unmapped address on the
   configuration above.
2. Make NVreg_DmaRemapPeerMmio readable at runtime. Its value decides whether peer MMIO is
   mapped at all, and it is currently unobservable — we had to reboot with it on the kernel
   command line just to establish what it was.
3. Consider a diagnostic on the fallback path. Today it succeeds silently and hands back an
   address the importer cannot use; nothing in the log or in any return code says so. Every
   driver that hits this will spend the same day we did finding it.

Related

Discussion #1046 describes the same shape of use case — an FPGA bus master writing into a
CUDA allocation through dma-buf — and is unanswered.

To Reproduce

Reproduce

User space, flags 0 (no CU_MEM_RANGE_FLAG_DMA_BUF_MAPPING_TYPE_PCIE):

cuMemGetHandleForAddressRange(&Fd, DevPtr, Size, CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0);

Importer, the ordinary sequence:

Buf     = dma_buf_get(Fd);
Attach  = dma_buf_attach(Buf, &pdev->dev);          /* the bus-mastering device */
SgTable = dma_buf_map_attachment_unlocked(Attach, DMA_FROM_DEVICE);
for_each_sgtable_dma_sg(SgTable, Sg, i)             /* sg_dma_address / sg_dma_len only */

A 64 MB allocation returns 1024 segments of 65536 bytes.

With iommu=pt:

seg 0: dma 0x41720000, 65536 bytes
seg 1: dma 0x41730000, 65536 bytes

Without iommu=pt, same machine, same run:

seg 0: dma 0x41720000, 65536 bytes   iommu_iova_to_phys() = 0
seg 1: dma 0x41730000, 65536 bytes   iommu_iova_to_phys() = 0

Identical addresses, and none resolve in the importing device's domain. 0x41720000 is
inside the GPU's BAR1 window. The device writes there and VT-d drops it:

dmar_fault: 1133498 callbacks suppressed
DMAR: [DMA Write NO_PASID] Request device [02:00.0] fault addr 0x41720000
      [fault reason 0x05] PTE Write access is not set

Every call succeeds. dma_buf_get, dma_buf_attach and dma_buf_map_attachment_unlocked
all return normally. The only symptom is that no data arrives.

Bug Incidence

Always

nvidia-bug-report.log.gz

Reproduce

User space, flags 0 (no CU_MEM_RANGE_FLAG_DMA_BUF_MAPPING_TYPE_PCIE):

cuMemGetHandleForAddressRange(&Fd, DevPtr, Size, CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0);

Importer, the ordinary sequence:

Buf     = dma_buf_get(Fd);
Attach  = dma_buf_attach(Buf, &pdev->dev);          /* the bus-mastering device */
SgTable = dma_buf_map_attachment_unlocked(Attach, DMA_FROM_DEVICE);
for_each_sgtable_dma_sg(SgTable, Sg, i)             /* sg_dma_address / sg_dma_len only */

A 64 MB allocation returns 1024 segments of 65536 bytes.

With iommu=pt:

seg 0: dma 0x41720000, 65536 bytes
seg 1: dma 0x41730000, 65536 bytes

Without iommu=pt, same machine, same run:

seg 0: dma 0x41720000, 65536 bytes   iommu_iova_to_phys() = 0
seg 1: dma 0x41730000, 65536 bytes   iommu_iova_to_phys() = 0

Identical addresses, and none resolve in the importing device's domain. 0x41720000 is
inside the GPU's BAR1 window. The device writes there and VT-d drops it:

dmar_fault: 1133498 callbacks suppressed
DMAR: [DMA Write NO_PASID] Request device [02:00.0] fault addr 0x41720000
      [fault reason 0x05] PTE Write access is not set

Every call succeeds. dma_buf_get, dma_buf_attach and dma_buf_map_attachment_unlocked
all return normally. The only symptom is that no data arrives.

More Info

No response

Start by comparing the 595.84 implementations of nv_dma_use_map_resource() and nv_dma_map_peer() with main, tracing which branch produces the unmapped BAR address. Review nv_dma_attach() and the nv_dma_remap_peer_mmio parameter handling, then use the dma-buf reproduction and IOMMU fault output to verify the result. Done means the responsible path is identified and the requested observability or diagnostic behavior has a clear implementation plan.