[FEA] Represent column labels in `ColumnAccessor` as a pandas Index instead of dictionary keys
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Currently `DataFrame.columns` is represented/stored as dictionary keys in `ColumnAccessor._data`. This works enough to match a fair amount of pandas functionality but not without some limitations and reinvention:
* `DataFrame` does not support duplicate column labels like pandas (xref https://github.com/rapidsai/cudf/issues/16533)
* We've needed to store extra metadata to capture the metadata of `DataFrame.columns` (like the class type, and data type) in various attributes (`ColumnAccessor.multiindex` , `ColumnAccessor.label_dtype`)
* `DataFrame.columns` does not correctly store the correct metadata for each level of a `MultiIndex` (xref https://github.com/rapidsai/cudf/issues/14267)
* We've needed to reinvent a lot of methods for column indexing via `DataFrame.__getitem__`, `DataFrame.iloc/loc` either in `DataFrame` or `ColumnAccessor`
**Describe the solution you'd like**
Since we already `DataFrame.columns` as host data today, I would propose we just utilize a `pandas.Index` to represent column labels instead of dictionary keys to address all the above points. Therefore have `ColumnAccessor` store
```python
class ColumnAccessor(abc.MutableMapping):
column_labels: pandas.Index
_data: list[ColumnBase]
```
To access columns via label, we utilize `column_labels.get_loc` (https://pandas.pydata.org/docs/reference/api/pandas.Index.get_loc.html) to get an indexer (`int`, `slice`, boolean mask) into `_data`, e.g.
```python
In [5]: _data = ["col1", "col2"]
In [6]: column_labels = pd.Index(["a", "b"])
In [7]: _data[column_labels.get_loc("a")]
Out[7]: 'col1'
```
Some potential downsides include:
* Increased host memory footprint for storing this pandas `Index`
* Single label column access would be slower
```python
In [11]: _data = {0: "col1", 1: "col2"}
In [12]: %timeit _data[0]
29 ns ± 0.184 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
In [13]: column_labels = pd.Index(["a", "b"])
In [14]: column_labels.get_loc("a") # best case, populate the Index hash table
Out[14]: 0
In [15]: _data = ["col1", "col2"]
In [16]: %timeit _data[column_labels.get_loc("a")]
274 ns ± 2.07 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```
**Describe alternatives you've considered**
Status quo
**Additional context**
Add any other context, code examples, or references to existing implementations about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.