NVIDIA / NVIDIA/cudf

[FEA]: Use `Shuffler.concat_insert` in cudf-polars

Open
#19,590 1 comment 1 reaction 0 assignees View on GitHub
cudf-polars feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

https://github.com/rapidsai/rapidsmpf/pull/361 added a `Shuffler.concat_insert`, which might offer better performance on insert. This issue will track exposing that in cudf-polars and benchmarking it:

- [ ] Find where to update: I guess this will actually be in `cudf-polars`, call `shuffler.concat_insert` instead of `shuffler.insert_chunks` at https://github.com/rapidsai/cudf/blob/5853debea4ae85b48b99e8f556973ea244f6a71a/python/cudf_polars/cudf_polars/experimental/shuffle.py#L72.
- [ ] Add a configuration option to control which is used at that point
- [ ] Run some benchmarks (the PDSH benchmarks that do shuffles). Get timings and nsys profiles for both insert types.
- [ ] Figure out what the default should be, whether it should be static or dynamic, and whether we should keep the configuration option (is one strictly better?)

**Describe the solution you'd like**

For now, a configuration option to choose the insert method.

**Describe alternatives you've considered**

N/A

**Additional context**

cc @nirandaperera, does that all sound right to you? Can I still just pass in the `packed_inputs` that's the result of `partition_and_pack` there?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.