[QST] Potential I/O Performance Improvement with cuFile Buffer Registration?

Hi cuIO Team,

I'm reaching out based on my profiling results. While examining the "CUDA HW" row during the `kvikio.cufileRead` operation in the Parquet reading process, I noticed numerous but tiny "Memcpy D2D" calls. As before, my workload consists of reading Parquet files from an SSD via GDS as #18278


# What are MemcpyD2D and cuFile Buffer Registration?

From GDS, these "Memcpy D2D" calls originate from a staging buffer on the GPU: https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html#cufilebufregister-cufileread-cufilewrite-cufilebatchiosubmit-cufilebatchiogetstatus-cufilereadasync-cufilewriteasync-and-cufilestreamregister

**cuFile memory registration** enables DMA without the need for this GPU staging buffer. With cuFile memory registration call, there will almost empty in "CUDA HW" row during I/O SSD reading.

# Are MemcpyD2D a performance bottleneck?

I don't believe these calls are a direct performance bottleneck. But the CPU has to launch a large number of MemcpyD2D kernels, which consume a portion of the GPU's memory bandwidth.

Within cuDF:
- conducting an experiment is challenging. It requires code changes to create a reusable I/O buffer destination for registration. Moreover, I doubt that optimizing this would yield a significant performance boost, and my pipeline hasn't been hindered by this issue so far

## Microbenchmarking

To test this, I only run a simple microbenchmark with kvikio. The workload involved reading a single file 100 times in single threaded and measuring the total time.


(I also conducted a similar test with GDSIO and obtained similar results, so I've omitted those details.)

# Short Conclusion

In summary, buffer registration doesn't seem to be time-consuming, and improving it won't result in a substantial performance increase.
- It's mainly beneficial when the number of KVIKIO threads is below 8, which is also below the default setting.

From an engineering perspective in libcudf, it could be a major undertaking.
- As in GDS buffer registration should not be part of the critical path, and registered buffers **should be reused** to amortize the cost.

After just three hours reading and microbenchmaring, so I'm presenting this as a topic **as a discussion** and would appreciate your insights. Please correct me if I'm wrong in my analysis.