dask / dask/dask-ml

OneHotEncoder with nan labels

Open
#926 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

Sklearn's OneHotEncoder supports `np.nan` values, including if they are passed as explicit categories:

```python
import numpy as np
from sklearn.preprocessing import OneHotEncoder

X = np.array([1, 2, 1, np.nan]).reshape(-1, 1)

version1 = OneHotEncoder(categories="auto").fit_transform(X).todense()
version2 = OneHotEncoder(categories=[[1, 2, np.nan]]).fit_transform(X).todense()
assert np.all(version1 == version2) == True
```

The `dask_ml` versions of the above will fail in both versions:

```python
import numpy as np
from dask.array import from_array
from dask_ml.preprocessing import OneHotEncoder

X = from_array(np.array([1, 2, 1, np.nan]).reshape(-1, 1))

version1 = OneHotEncoder(categories="auto").fit_transform(X).compute()
# fails with ValueError: Block contains previously unseen values [nan].

version2 = OneHotEncoder(categories=[[1, 2, np.nan]]).fit_transform(X).compute()
# fails with ValueError: Unsorted categories are not yet supported
```

It seems that at some point scikit learn allowed nans (which seems reasonable). For the last dask ml error the error is due to `np.nan` not being equal to itself:

```python
import numpy as np

cats = [1, 2, np.nan]

print(np.sort(cats) == np.sort(cats))
# [ True True False]
```

This happens here: https://github.com/dask/dask-ml/blob/67d28b15dfff7869e9a04def203aa129a3540b27/dask_ml/preprocessing/_encoders.py#L182-L190

**Environment**:

- Dask version: 2022.01.0
- Dask_ml: 2022.1.22
- Python version: 3.9
- Operating System: MacOS
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.