Categorizer does not preserve order of categories for Pandas != 1.2
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
For Pandas<1.2, the Categorizer does not always preserve the order of categories (due to a bug in `pd.series.astype`, see e.g., https://github.com/pandas-dev/pandas/issues/30206).
**Example:**
```python
import pandas as pd
from dask_ml.preprocessing import Categorizer
X1 = pd.DataFrame({"x": pd.Categorical(["a"], categories=["a", "b"])})
X2 = pd.DataFrame({"x": pd.Categorical(["a"], categories=["b", "a"])})
categorizer = Categorizer().fit(X1)
categorizer.transform(X1)["x"].dtype
# > CategoricalDtype(categories=['a', 'b'], ordered=False)
categorizer.transform(X2)["x"].dtype
# > CategoricalDtype(categories=['b', 'a'], ordered=False)
```
For Pandas>=1.2, the above code snippet produces the same result for `X1` and `X2` (as we would expect).
This behavior is caused by this call to `pd.series.astype`:
https://github.com/dask/dask-ml/blob/0ea276da1d78db582f40e1c256dfca4f70e6cbc6/dask_ml/preprocessing/data.py#L568
**Pandas-only example:**
```python
x1 = pd.Series(pd.Categorical(["a"], categories=["a", "b"]))
x2 = pd.Series(pd.Categorical(["a"], categories=["b", "a"]))
x2.astype(x1.dtype)
# > 0 a
# > dtype: category
# > Categories (2, object): ['b', 'a']
```
Again, I would expect that astype enforces the order (but that only happens for pandas>=1.2).
**Question**
Is it worth fixing this for Pandas<1.2? This can cause issues for downstream estimators where the order of categories matters (e.g. because they're used for one-hot encoding of some sort). I would argue that Pandas<1.2 is still pretty common.
I'd be happy to contribute a fix.
Another question is if one should ever rely on the order of categories in Pandas categorical types...
**Environment**:
- Dask version: 2021.4.0
- Python version: 3.8.8
- Operating System: osx
- Install method (conda, pip, source): source (1.8.1.dev19+g0ea276da)
Contributor guide
Research direction
Start in dask_ml/preprocessing/data.py around line 568 and reproduce the provided Categorizer examples with a Pandas version below 1.2. Preserve the fitted category order during transformation and add a regression test covering X1 and X2, including the expected categorical dtypes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100