NVIDIA / NVIDIA/cudf

[FEA] Serialization of libcudf classes and exposing implementation details

Open
#17,630 4 comments 1 reaction 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
For multi-gpu polars, we will require serializing certain data in Python to be passed between Dask workers, for example `aggregation`s. In https://github.com/rapidsai/cudf/pull/17469 I've proposed a way to do that, however, that proposal requires certain implementation details from [`aggreation.hpp`](https://github.com/rapidsai/cudf/blob/branch-25.02/cpp/include/cudf/detail/aggregation/aggregation.hpp),
more specifically classes derived from `aggregation`, such as [`std_var_aggregation`](https://github.com/rapidsai/cudf/blob/0e01dbd2d8b2a048964d27d1fd9d9c3bf6540166/cpp/include/cudf/detail/aggregation/aggregation.hpp#L384-L413). @vyasr has pointed out to the fact that those details are not exposed to pylibcudf and would be best if it continues like that.

**Describe the solution you'd like**
The solution proposed in https://github.com/rapidsai/cudf/pull/17469 seems to be the lowest hanging fruit, but as described above may not be considered optimal for several reasons.

**Describe alternatives you've considered**
Exposing attributes of the classes publicly may be an alternative, but that would incur in a different set of potential issues.

I'm not familiar with most of the design and options available in libcudf, so it's likely core developers will see other potentially better alternatives.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.