[FEA] Test a fallback MR from async to pinned host in cudf-polars
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
As of 25.12, cudf-polars uses a managed memory resource by default. The full MR is a prefetch MR wrapping a pool MR wrapping a managed MR.
Soon we will have a new MR pattern available:
```
new_mr = rmm.mr.FallbackResourceAdaptor(
rmm.mr.CudaAsyncMemoryResource(),
rmm.mr.PoolMemoryResource(
rmm.mr.HostPinnedMemoryResource(
[XX% of host memory]
)
)
)
```
Depends on:
https://github.com/rapidsai/rmm/issues/2090
https://github.com/rapidsai/rmm/issues/1429
https://github.com/rapidsai/rmm/issues/2074
This MR could outperform managed memory in some cases. It would not encounter prefetch thrashing, where prefetching data also evicts data needed by subsequent kernels. It's possible that there is a performance boost when running kernels against pinned host memory versus delaying kernel launch until prefetching is complete. This MR would also have no prefetching overhead when undersubscribed, and would be equivalent to an async MR until the first OOM.
The downside of this approach would be fragmentation of the pinned pool, but hopefully the fragmentation can be reset whenever it empties.
**Additional context**
We've seen some indications from performance testing in Velox-cuDF that managed memory with prefetching shows some performance issues when there are multiple workers per GPU. The prefetch-on-allocate MR uses a non-default stream for prefetching, and this should not interrupt pipelining. However, at least some of the prefetch-on-access calls in cuDF use the default stream, and this could interrupt pipelining.
We will need to study managed memory with multiple drivers and when undersubscribed to assess if poor pipelining is the source of performance issues in Velox-cuDF. We will also see how managed memory performs when cudf-polars gains the capability to run multi-threaded.
Update: maybe using managed memory with PTDS per-thread-default-stream will prevent issues with prefetches blocking pipelining!!
Contributor guide
Assessment
This issue has not been assessed yet.