Consistent Updating of LabelEncoder Dtype with the "Fit" Interfaces
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
#### Problem
LabelEncoder updates it's internal Dtype `clazz.dtype_` in different ways; dependent on the class method used to fit the Encoder. Fitting a `dask.dataframe.Series` with `fit_transform()` makes it impossible to subsequently transform the very same series.
*Side Note/Anecdote: as a developer I want to use arbitrary Encoders across my datasets (train, validation, test, other holdouts). I want to be able to train an encoder using the train dataset and then apply that encoding to the other datasets. If information (like a symbol to encode) isn't found in these other datasets, that information should be redacted (using an unknown symbol). I get the feel that dask LabelEncoders weren't designed in this way. I think this gives rise to the problem above.
#### Problem in the `dask_ml` Library
`dask_ml.preprocessing.LabelEncoder.fit()` uses this [code](https://github.com/dask/dask-ml/blob/master/dask_ml/preprocessing/label.py#L120)
```python
elif _is_categorical(y):
self.classes_ = _encode_categorical(y)
self.dtype_ = y.dtype
```
`dask_ml.preprocessing.LabelEncoder.fit_transform()` uses this [code](https://github.com/dask/dask-ml/blob/master/dask_ml/preprocessing/label.py#L135):
```python
elif _is_categorical(y):
self.classes_, y = _encode_categorical(y, encode=True) # the problem
self.dtype_ = y.dtype
```
#### Sample Code
```python
import dask
import dask_ml
categorical_cols = ['x']
df = dask.dataframe.from_pandas(
pd.DataFrame(list(map(str, range(100))), columns=categorical_cols),
npartitions=1
)
df = df.categorize(columns=categorical_cols)
le = dask_ml.preprocessing.LabelEncoder()
_ = le.fit_transform(df[categorical_cols[0]])
print(f'Dtype should be CategoricalDtype. Dtype was encoded to {le.dtype_}')
print('''
Transforming the same dd.Series will fail with the Traceback:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in
12 print(f'Dtype should be CategoricalDtype. Dtype was encoded to {le.dtype_}')
13 print(f'Transforming the same dd.Series will fail')
---> 14 le.transform(df[categorical_cols[0]])
~/conda_envs/ml_res_cancellations/lib/python3.6/site-packages/dask_ml/preprocessing/label.py in transform(self, y)
147 return _encode_dask_array(y, self.classes_, encode=True)[1]
148 elif isinstance(y, (pd.Series, dd.Series)):
--> 149 assert y.dtype.categories.equals(self.dtype_.categories)
150 return y.cat.codes.values
151 else:
AttributeError: 'numpy.dtype' object has no attribute 'categories'
''')
# fail
le.transform(df[categorical_cols[0]])
```
Contributor guide
Assessment
This issue has not been assessed yet.