[FEA][INTERNALS] A `ColumnMeta` type to represent the column metadata of a `Frame`
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
When we roundtrip a `Frame` between Python and libcudf, we potentially lose a bunch of metadata:
1. Names of columns
2. Whether the columns have multiple levels (i.e., the Frame has a MultiIndex as its columns(
3. The level names
## The problem
libcudf functions return a `unique_ptr`, we convert that `table` into a `Frame` in the function [from_unique_ptr](https://github.com/rapidsai/cudf/blob/ec5364c2fc0a3c63583e648ec90efa8d3b5675bc/python/cudf/cudf/_lib/table.pyx#L82). Here, we pass the column names (1), but not the multiindex (2) or level_names (3) metadata.
This can lead to surprising behaviour in many situations. For example, consider the `loc` call below where we lose the `multiindex` part of our metadata:
```python
In [10]: df
Out[10]:
a b
sum min max min
a
2 4 2 5 4
1 3 1 3 1
In [11]: df.loc[[2, 2, 1, 1], :]
Out[11]:
(a, sum) (a, min) (b, max) (b, min)
a
2 4 2 5 4
2 4 2 5 4
1 3 1 3 1
1 3 1 3 1
In [12]: df.to_pandas().loc[[2, 2, 1, 1], :]
Out[12]:
a b
sum min max min
a
2 4 2 5 4
2 4 2 5 4
1 3 1 3 1
1 3 1 3 1
```
## Proposed solution
We could introduce an internal `ColumnMeta` type:
```python
class ColumnMeta:
names: Tuple[Any]
multiindex: bool
level_names: Optional[Tuple[Any]]
```
which could be a property of `Frame` objects for convenience:
```python
class Frame:
@cached_property
def _column_meta(self):
...
```
Now, instead of passing just the column names and index names to `from_unique_ptr`, we could pass the full metadata for both:
```python
cdef Table from_unique_ptr(
unique_ptr[table] c_tbl,
ColumnMeta data_meta,
ColumnMeta index_meta=None
):
```
and it would construct the resulting `Frame` with the correct column metadata.
--
With this, a typical Python wrapper around a libcudf API would be:
```python
def py_func(Table foo, ...):
cdef table_view c_input = foo.view()
cdef unique_ptr[table] c_result
with nogil:
c_result = cpp_func(c_input)
return Table.from_unique_ptr(c_result, foo._column_meta, foo.index._column_meta)
```
Contributor guide
Assessment
This issue has not been assessed yet.