dask / dask/dask-ml

Categorizer does not preserve order of categories for Pandas != 1.2

Open
#829 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

For Pandas<1.2, the Categorizer does not always preserve the order of categories (due to a bug in `pd.series.astype`, see e.g., https://github.com/pandas-dev/pandas/issues/30206).

**Example:**

```python
import pandas as pd
from dask_ml.preprocessing import Categorizer

X1 = pd.DataFrame({"x": pd.Categorical(["a"], categories=["a", "b"])})
X2 = pd.DataFrame({"x": pd.Categorical(["a"], categories=["b", "a"])})

categorizer = Categorizer().fit(X1)

categorizer.transform(X1)["x"].dtype
# > CategoricalDtype(categories=['a', 'b'], ordered=False)

categorizer.transform(X2)["x"].dtype
# > CategoricalDtype(categories=['b', 'a'], ordered=False)
```

For Pandas>=1.2, the above code snippet produces the same result for `X1` and `X2` (as we would expect).

This behavior is caused by this call to `pd.series.astype`:

https://github.com/dask/dask-ml/blob/0ea276da1d78db582f40e1c256dfca4f70e6cbc6/dask_ml/preprocessing/data.py#L568

**Pandas-only example:**
```python
x1 = pd.Series(pd.Categorical(["a"], categories=["a", "b"]))
x2 = pd.Series(pd.Categorical(["a"], categories=["b", "a"]))
x2.astype(x1.dtype)
# > 0 a
# > dtype: category
# > Categories (2, object): ['b', 'a']
```
Again, I would expect that astype enforces the order (but that only happens for pandas>=1.2).

**Question**

Is it worth fixing this for Pandas<1.2? This can cause issues for downstream estimators where the order of categories matters (e.g. because they're used for one-hot encoding of some sort). I would argue that Pandas<1.2 is still pretty common.

I'd be happy to contribute a fix.

Another question is if one should ever rely on the order of categories in Pandas categorical types...

**Environment**:

- Dask version: 2021.4.0
- Python version: 3.8.8
- Operating System: osx
- Install method (conda, pip, source): source (1.8.1.dev19+g0ea276da)

Contributor guide

Open the contributing guide

Research direction

Start in dask_ml/preprocessing/data.py around line 568 and reproduce the provided Categorizer examples with a Pandas version below 1.2. Preserve the fitted category order during transformation and add a regression test covering X1 and X2, including the expected categorical dtypes.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.