NVIDIA / NVIDIA/cudf

[FEA] Represent column labels in `ColumnAccessor` as a pandas Index instead of dictionary keys

Open
#18,905 0 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
Currently `DataFrame.columns` is represented/stored as dictionary keys in `ColumnAccessor._data`. This works enough to match a fair amount of pandas functionality but not without some limitations and reinvention:

* `DataFrame` does not support duplicate column labels like pandas (xref https://github.com/rapidsai/cudf/issues/16533)
* We've needed to store extra metadata to capture the metadata of `DataFrame.columns` (like the class type, and data type) in various attributes (`ColumnAccessor.multiindex` , `ColumnAccessor.label_dtype`)
* `DataFrame.columns` does not correctly store the correct metadata for each level of a `MultiIndex` (xref https://github.com/rapidsai/cudf/issues/14267)
* We've needed to reinvent a lot of methods for column indexing via `DataFrame.__getitem__`, `DataFrame.iloc/loc` either in `DataFrame` or `ColumnAccessor`

**Describe the solution you'd like**
Since we already `DataFrame.columns` as host data today, I would propose we just utilize a `pandas.Index` to represent column labels instead of dictionary keys to address all the above points. Therefore have `ColumnAccessor` store

```python
class ColumnAccessor(abc.MutableMapping):
column_labels: pandas.Index
_data: list[ColumnBase]
```

To access columns via label, we utilize `column_labels.get_loc` (https://pandas.pydata.org/docs/reference/api/pandas.Index.get_loc.html) to get an indexer (`int`, `slice`, boolean mask) into `_data`, e.g.

```python
In [5]: _data = ["col1", "col2"]

In [6]: column_labels = pd.Index(["a", "b"])

In [7]: _data[column_labels.get_loc("a")]
Out[7]: 'col1'
```

Some potential downsides include:

* Increased host memory footprint for storing this pandas `Index`
* Single label column access would be slower

```python
In [11]: _data = {0: "col1", 1: "col2"}

In [12]: %timeit _data[0]
29 ns ± 0.184 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [13]: column_labels = pd.Index(["a", "b"])

In [14]: column_labels.get_loc("a") # best case, populate the Index hash table
Out[14]: 0

In [15]: _data = ["col1", "col2"]

In [16]: %timeit _data[column_labels.get_loc("a")]
274 ns ± 2.07 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```

**Describe alternatives you've considered**
Status quo

**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.