NVIDIA / NVIDIA/cudf

[FEA] Support aggregations/scans on lists via groupby

Open
#13,208 0 comments 1 reaction 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

This is related to https://github.com/rapidsai/cudf/issues/10408.

It's possible to use a combination of `explode()` and `groupby()` to support common aggregations on `list` columns:

```python
In [23]: df
Out[23]:
a
0 [1, 3, 2]
1 [1, 4]
2 [9, 0, -1]

In [24]: df.explode('a').groupby(level=0).max()
Out[24]:
a
0 3
1 4
2 9
```

Scans can similarly be computed via an additional call to `groupby-collect()`:

```python
In [33]: df.explode('a').groupby(level=0).cummax().groupby(level=0).collect()
Out[33]:
a
0 [1, 3, 3]
1 [1, 4]
2 [9, 9, 9]
```

When an aggregation/scan supported by groupby, like `.max()` or `.min()` is called on a list column, we could transparently use this combination of explode + groupby to support that operation. That could look something like:

```python
>>> df
a
0 [1, 3, 2]
1 [1, 4]
2 [9, 0, -1]

>>> df['a'].list.max()
0 3
1 4
2 9
Name: a, dtype: int64

>>> df['a'].list.cummax()
0 [1, 3, 3]
1 [1, 4]
2 [9, 9, 9]
Name: a, dtype: list
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.