rdma: GPU-Direct throughput anti-scales across processes on one host

Summary

GPU-Direct RDMA aggregate throughput decreases as more processes on the same host use it. Eight processes together move less than one process does. This is the main thing stopping S3-over-RDMA from being usable for multi-GPU DDP training, where one process per GPU is the norm.

I don't think this is a minio-cpp bug — it looks like it sits in libcuobjclient/libcufile or in the server's per-client RDMA resource handling — but filing here since this is where the client stack lives and you'll know who to route it to.

Measurement

Client: 8× H200, 256 cores, single 400 GbE RoCE rail (iperf3 ceiling 41.5 GB/s).
Server: 2-node AIStor RELEASE.2026-08-07T18-34-35Z, 48 NVMe, EC:4.
minio-cpp: main @ e764b82, built -DMINIO_CPP_ENABLE_RDMA=ON.
Each process runs 8 threads, each thread with its own 768 MiB CUDA destination buffer, reading 768 MiB objects.

At 256 MiB buffers the RDMA share stays 100% at 8 processes and it is still slow, so the fallback and the slowdown are two separate effects:

Aggregate operation rate collapses too, roughly independently of size: 139 ops/s at 1 process vs 17.5 ops/s at 8 (64 MiB objects).

What I ruled out

• ibverbs resource limits. max_mr = 16,777,216, max_pd = 8,388,608, max_mr_size unlimited; only ~374 MRs in use.
• cuFile pinned-memory cap. Setting max_device_pinned_mem_size_kb to 64 GiB explicitly changed nothing.
• Per-transfer registration churn. I prototyped a caller-pinned buffer registry so cuMemObjGetDescriptor/cuMemObjPutDescriptor are skipped for pre-pinned buffers. It made no reliable difference (median 9.77 vs 10.02 GB/s at 256 MiB — if anything slightly worse), so I dropped it rather than propose it. Registration churn is not the mechanism.
• A crash. Earlier builds segfaulted in this shape; main does not. This is now purely a throughput/fallback problem.

Note on the silent fallback

At 768 MiB × 8 processes only a third of the bytes were carried by RDMA; the rest completed over HTTP with no error surfaced. Since use_rdma is just cuMemObjGetDescriptor(...) == 0, a registration that fails under pressure is indistinguishable from "RDMA not wanted", and the caller sees a successful transfer that was three times slower than it should have been.

It would help a lot if a declined-because-registration-failed transfer were distinguishable from a normal HTTP fallback — a counter, or a log at debug level. Right now the only way to tell is to diff minio_api_rdma_read_bytes_total on the server across the operation, which is what I ended up doing.

Reproduction

Harness and full raw results (JSON per run) are in a benchmark repo I can share if useful. The shape is: N processes, each with 8 threads, each thread issuing miniocpp_get_object into its own registered CUDA buffer, with the server's minio_api_rdma_read_bytes_total sampled around the run to compute the RDMA share.

Start with the RDMA-enabled minio-cpp path built using MINIO_CPP_ENABLE_RDMA and the miniocpp_get_object calls described in the reproduction. Inspect how cuMemObjGetDescriptor determines use_rdma and compare client behavior with the server's minio_api_rdma_read_bytes_total metric. Done requires identifying whether the throughput regression or silent fallback is actionable in this repository, or routing it to the relevant cuFile/libcuobjclient or server component.