NVIDIA / NVIDIA/cudf

[FEA] Allow groupby scan aggregations to return listified results

Open
#15,541 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

To match the way scan aggregation results in groupby operations are returned in pandas, libcudf returns scan-based aggregations in the same shape as the input table (these are then optionally reordered to mimic pandas order).

For the cudf-polars executor, it would be useful to also have a mode where the result of a scan aggregation is collected, group-wise, into a list column. This would mean that both scan-like and reduction-like groupby aggregations always produce an output table with a number of rows equal to the number of unique group keys.

**Describe the solution you'd like**

For scan-only aggregations, this is relatively easy to achieve by taking the sorted grouped result and calling `make_lists_column` with the group offsets. When mixing scan and hash-based aggregations it is tricker (since those would spit things out in a different order and would then need a join). Ideally one the scan aggs have a "collect as list" option, then one would be able to do scan and reduce- aggs in the same call on the sorted table.

**Describe alternatives you've considered**

I can post-process the result (and then do a join if I have any hash-based aggs in addition).

**Additional context**

Right now, polars guarantees that although the order of groups in the result is implementation dependent, within a group, the rows show up in original dataframe order. I think this is also guaranteed by libcudf, since the sort-by-key before the aggregations is stable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.