OneHotEncoder with nan labels
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
Sklearn's OneHotEncoder supports `np.nan` values, including if they are passed as explicit categories:
```python
import numpy as np
from sklearn.preprocessing import OneHotEncoder
X = np.array([1, 2, 1, np.nan]).reshape(-1, 1)
version1 = OneHotEncoder(categories="auto").fit_transform(X).todense()
version2 = OneHotEncoder(categories=[[1, 2, np.nan]]).fit_transform(X).todense()
assert np.all(version1 == version2) == True
```
The `dask_ml` versions of the above will fail in both versions:
```python
import numpy as np
from dask.array import from_array
from dask_ml.preprocessing import OneHotEncoder
X = from_array(np.array([1, 2, 1, np.nan]).reshape(-1, 1))
version1 = OneHotEncoder(categories="auto").fit_transform(X).compute()
# fails with ValueError: Block contains previously unseen values [nan].
version2 = OneHotEncoder(categories=[[1, 2, np.nan]]).fit_transform(X).compute()
# fails with ValueError: Unsorted categories are not yet supported
```
It seems that at some point scikit learn allowed nans (which seems reasonable). For the last dask ml error the error is due to `np.nan` not being equal to itself:
```python
import numpy as np
cats = [1, 2, np.nan]
print(np.sort(cats) == np.sort(cats))
# [ True True False]
```
This happens here: https://github.com/dask/dask-ml/blob/67d28b15dfff7869e9a04def203aa129a3540b27/dask_ml/preprocessing/_encoders.py#L182-L190
**Environment**:
- Dask version: 2022.01.0
- Dask_ml: 2022.1.22
- Python version: 3.9
- Operating System: MacOS
- Install method (conda, pip, source): conda
Contributor guide
Assessment
This issue has not been assessed yet.