h2oai / h2oai/datatable

Implementation of Categorical Columns in datatable

Open
#3,136 0 comments 1 reaction 1 assignee Claimed by @oleksiyskononenko View on GitHub
design-doc
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

This is a short design doc on implementing categorical columns in datatable, as it is proposed in #1691. This document is going to be modified while we go through the implementation stage and have a better understanding of the details.

### Brief Overview

- In pandas there is a [Categorical](https://pandas.pydata.org/docs/reference/api/pandas.Categorical.html) data type, for which there exists a comprehensive [introduction](https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html);
- In Apache Arrow there is a similar [Dictionary](https://arrow.apache.org/docs/python/data.html#dictionary-arrays) type;
- In R there is a function [factor()](https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/factor) to encode a vector as a factor;
- in Julia there exists a [CategoricalArray](https://dataframes.juliadata.org/stable/man/categorical/).

### Proposed Implementation in datatable

In datatable we should add several new types, that could be called `dt.Type.cat{N}(T)`, where `N` is `8`, `16`, `32` or `64`. The corresponding `Categorical_ColumnImpl` will carry an array of `codes`, that could be implemented as a `Buffer`, and a Column of `categories`, which could have arbitrary `Type`.

For a user, categorical columns should be made transparent, i.e. when accessing their content, categorical columns would behave like normal `ColumnImpl` columns of a type `T`. Categories could be accessible through a special function, for instance `dt.cat.categories()`.

Updating content of the categorical column is a little bit tricky, because then in some cases one needs to update the underlying `ColumnImpl` with a new `category`. If a category already exists, then only the array of `codes` should be updated.

### Creating Categorical Columns

It should be possible to create categorical columns by providing data and a list of categories. An indicator of whether the categories are ordered could be kept as part of the corresponding `dt.Type.cat{N}(T)`. For instance, something like this should work:

```python

DT = dt.Frame(MyCategoricalColumn = dt.cat.from(["a", "b", "a"], categories = ["a", "b"], ordered = False))

```

### Operations on Categorical Columns

Any operation allowed on the type `T` could also be allowed on the corresponding `dt.Type.cat{N}(T)`, however,
in this case the result of the operation will not be a categorical column anymore.

At the same time, rbinding categorical columns of the same type `T` could preserve the categorical type.

Some other basic operations on categories should be available:
- adding new categories;
- renaming existing categories;
- removing existing categories;
- merging categories, for instance, in the case of `rbind`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.