NVIDIA / NVIDIA/cudf

[FEA] Get Series.list offsets / Construct Series of lists from offsets and values

Open
#11,130 15 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
I would like to be able to access the offsets of a Series of lists. That would allow me to implement a function like `list_add` that takes two "awkward arrays," Series of lists of numbers that have the same list shape, and adds them together. The binary operation can be straightforwardly applied to the "leaves" of each list column, which is the child column containing the data. However, to do this, I need a way to access indices and rebuild the list structure. For example, if `Series.list.offsets` and `cudf.Series.list.from_arrays(offsets, values)` existed, I could run something like:

```python
def list_add(s1, s2):
"""Take two Series of lists of numerical data and add them."""
# Ignore nested lists for simplicity -- this only works for a single level of lists
if s1.list.offsets!= s2.list.offsets:
raise ValueError("List columns must have corresponding offsets.")
return cudf.Series.list.from_arrays(s1.list.offsets, s1.list.leaves + s2.list.leaves)
```

**Describe the solution you'd like**
- Implement a property `Series.list.offsets` that exposes the offset array, similar to [PyArrow's `pyarrow.ListArray.offsets`](https://arrow.apache.org/docs/python/generated/pyarrow.ListArray.html#pyarrow.ListArray.offsets) but returning a GPU-resident array.

- Implement a constructor `Series.list.from_arrays(offsets, values)` that builds a Series of lists from input offsets and values, similar to [PyArrow's `pyarrow.ListArray.from_arrays`](https://arrow.apache.org/docs/python/generated/pyarrow.ListArray.html#pyarrow.ListArray.from_arrays) but enabling construction from GPU-resident arrays.

**Describe alternatives you've considered**
I strongly prefer this approach over implementing binops directly on list types because it allows for precise control of what APIs are exposed and how they behave. Implementing binops for lists would allow for operators like `+` to be used, which is prone to error because it overloads the Python-like list semantics of "adding is list concatenation" with the array-like semantics of normal addition.

**Additional context**
It's not clear to me where the name "leaves" came from. To align with PyArrow, we would rename "leaves" to `Series.list.values`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.