NVIDIA / NVIDIA/cudf

[FEA][INTERNALS] A `ColumnMeta` type to represent the column metadata of a `Frame`

Open
#7,644 3 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

When we roundtrip a `Frame` between Python and libcudf, we potentially lose a bunch of metadata:

1. Names of columns
2. Whether the columns have multiple levels (i.e., the Frame has a MultiIndex as its columns(
3. The level names

## The problem

libcudf functions return a `unique_ptr`, we convert that `table` into a `Frame` in the function [from_unique_ptr](https://github.com/rapidsai/cudf/blob/ec5364c2fc0a3c63583e648ec90efa8d3b5675bc/python/cudf/cudf/_lib/table.pyx#L82). Here, we pass the column names (1), but not the multiindex (2) or level_names (3) metadata.

This can lead to surprising behaviour in many situations. For example, consider the `loc` call below where we lose the `multiindex` part of our metadata:

```python
In [10]: df
Out[10]:
a b
sum min max min
a
2 4 2 5 4
1 3 1 3 1

In [11]: df.loc[[2, 2, 1, 1], :]
Out[11]:
(a, sum) (a, min) (b, max) (b, min)
a
2 4 2 5 4
2 4 2 5 4
1 3 1 3 1
1 3 1 3 1

In [12]: df.to_pandas().loc[[2, 2, 1, 1], :]
Out[12]:
a b
sum min max min
a
2 4 2 5 4
2 4 2 5 4
1 3 1 3 1
1 3 1 3 1
```

## Proposed solution

We could introduce an internal `ColumnMeta` type:

```python
class ColumnMeta:
names: Tuple[Any]
multiindex: bool
level_names: Optional[Tuple[Any]]
```

which could be a property of `Frame` objects for convenience:

```python
class Frame:
@cached_property
def _column_meta(self):
...
```

Now, instead of passing just the column names and index names to `from_unique_ptr`, we could pass the full metadata for both:

```python
cdef Table from_unique_ptr(
unique_ptr[table] c_tbl,
ColumnMeta data_meta,
ColumnMeta index_meta=None
):
```

and it would construct the resulting `Frame` with the correct column metadata.

--

With this, a typical Python wrapper around a libcudf API would be:

```python
def py_func(Table foo, ...):
cdef table_view c_input = foo.view()
cdef unique_ptr[table] c_result
with nogil:
c_result = cpp_func(c_input)
return Table.from_unique_ptr(c_result, foo._column_meta, foo.index._column_meta)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.