NVIDIA / NVIDIA/cudf

[Story] Unify categorical support and implementation across engines

Open
#18,907 0 comments 0 reactions 0 assignees View on GitHub
feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Currently the different components of cudf handle categorical data to different degrees and in different ways:
- cuDF classic maintains a categorical column as a column one one child that is a numerical column. The categories are stored in the dtype.
- cudf.pandas uses cuDF classic under the hood
- cudf-polars does not currently support categoricals
- Spark RAPIDS does not directly support categorical data as its own type
- libcudf supports storing data using [the Arrow dictionary type](https://arrow.apache.org/docs/format/Columnar.html#dictionary-encoded-layout), but these have seen very limited use and the set of support operations is also limited

Ideally, we would unify all of these and use a single representation. #8573 captures this for cuDF classic, specifically. However, now that we are supporting multiple front ends with zero-code-change as the goal, we need to consider what each of the corresponding host tools do.

- pandas [supports both ordered and unordered categoricals](https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html). This feature is part of what has made it challenging to use the libcudf dictionary type in the past because that type is strictly unordered.
- [Polars supports two types of categoricals, Enum and Categorical](https://docs.pola.rs/user-guide/expressions/categorical-data-and-enums/). Under the hood, though, Polars also does not use the Arrow dictionary type in this case. Instead, Polars stores categorical data in an integer column. The categories are stored in the data type, while the column just contains the numerical values.
- Spark has no intrinsic concept of categoricals, so similar mappings to what Polars maintains internally must be maintained explicitly by the user

#8573 was written under the assumption that using libcudf's Dictionary type would lead to more performant and flexible code at the Python layer. With Polars also deviating from the Arrow type, however, it seems like we may need to go a different route anyway and define a custom categorical type based on numerical encodings that we can reuse across pandas and Polars. We should consider whether this approach would be preferable, and if so, close #8573. If we were to go this route, we would probably want to implement this categorical type using pylibcudf as an extension to pylibcudf's core and make that reusable for cudf.pandas and cudf-polars.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.