Implementation of Categorical Columns in datatable
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
This is a short design doc on implementing categorical columns in datatable, as it is proposed in #1691. This document is going to be modified while we go through the implementation stage and have a better understanding of the details.
### Brief Overview
- In pandas there is a [Categorical](https://pandas.pydata.org/docs/reference/api/pandas.Categorical.html) data type, for which there exists a comprehensive [introduction](https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html);
- In Apache Arrow there is a similar [Dictionary](https://arrow.apache.org/docs/python/data.html#dictionary-arrays) type;
- In R there is a function [factor()](https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/factor) to encode a vector as a factor;
- in Julia there exists a [CategoricalArray](https://dataframes.juliadata.org/stable/man/categorical/).
### Proposed Implementation in datatable
In datatable we should add several new types, that could be called `dt.Type.cat{N}(T)`, where `N` is `8`, `16`, `32` or `64`. The corresponding `Categorical_ColumnImpl` will carry an array of `codes`, that could be implemented as a `Buffer`, and a Column of `categories`, which could have arbitrary `Type`.
For a user, categorical columns should be made transparent, i.e. when accessing their content, categorical columns would behave like normal `ColumnImpl` columns of a type `T`. Categories could be accessible through a special function, for instance `dt.cat.categories()`.
Updating content of the categorical column is a little bit tricky, because then in some cases one needs to update the underlying `ColumnImpl` with a new `category`. If a category already exists, then only the array of `codes` should be updated.
### Creating Categorical Columns
It should be possible to create categorical columns by providing data and a list of categories. An indicator of whether the categories are ordered could be kept as part of the corresponding `dt.Type.cat{N}(T)`. For instance, something like this should work:
```python
DT = dt.Frame(MyCategoricalColumn = dt.cat.from(["a", "b", "a"], categories = ["a", "b"], ordered = False))
```
### Operations on Categorical Columns
Any operation allowed on the type `T` could also be allowed on the corresponding `dt.Type.cat{N}(T)`, however,
in this case the result of the operation will not be a categorical column anymore.
At the same time, rbinding categorical columns of the same type `T` could preserve the categorical type.
Some other basic operations on categories should be available:
- adding new categories;
- renaming existing categories;
- removing existing categories;
- merging categories, for instance, in the case of `rbind`.
Contributor guide
Assessment
This issue has not been assessed yet.