NVIDIA / NVIDIA/cudf

[FEA][JNI] Adopt rmm::pinned_host_memory_resource in java

Open
#14,782 0 comments 0 reactions 0 assignees View on GitHub
feature request Java Performance Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

RMM has introduced a pinned host memory resource, and is in the process to make changes for the pool memory resource to be compatible (https://github.com/rapidsai/rmm/pull/1392).

I have tested gutting `PinnedMemoryPool`, turning into a thin wrapper to a `pool_memory_resource` in a draft PR, that uses some initial work that @jbrennan333 started. Here's what we have so far: https://github.com/rapidsai/cudf/compare/branch-24.02...abellina:cudf:ab-pinned-pool.

We would like to make architecture decisions on how we could use the pinned pool, perhaps the single one created for the JNI side of things, to be shared with cuIO https://github.com/rapidsai/cudf/issues/14314.

It would be great or us to try and repro this https://github.com/rapidsai/cudf/issues/12341. And see if the new approach removes the slowness we have occasionally seen.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.