LabelEncoder raises errors with string and string[pyarrow] types
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
Using the label encode with `string` and `string[pyarrow]` dtypes raises TypeErrors. The error is getting raised in dask.
```
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/array/utils.py", line 57, in meta_from_array
x = x(shape=(0,) * (ndim or 0), dtype=dtype)
TypeError: Cannot interpret 'string[pyarrow]' as a data type
```
...But I can't trigger this without the LabelEncoder.
**Minimal Complete Verifiable Example**:
```python
import dask.dataframe as dd
from dask_ml.preprocessing import LabelEncoder
import pandas as pd
df = pd.DataFrame({
"A": list("aaaabbbcccdddeeefffgggg")
})
ddf = dd.from_pandas(df, npartitions=3)
# works
lenc = LabelEncoder().fit(ddf["A"])
# TypeError: Cannot interpret 'string[pyarrow]' as a data type
lenc = LabelEncoder().fit(ddf["A"].astype("string[pyarrow]"))
```
**Full Traceback**:
Very similar tracebacks for `string` and `string[pyarrow]`
➜ python label_encoder_repro.py
Traceback (most recent call last):
File "/Users/paul/work/sources/dask-engineering/example-pipelines/criteo-HPO/label_encoder_repro.py", line 12, in
lenc = LabelEncoder().fit(ddf["A"].astype("string[pyarrow]"))
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask_ml/preprocessing/label.py", line 115, in fit
y = self._check_array(y)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask_ml/preprocessing/label.py", line 111, in _check_array
y = y.to_dask_array(lengths=True)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 1687, in to_dask_array
arr = self.values
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 3431, in values
return self.map_partitions(methods.values)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 874, in map_partitions
return map_partitions(func, self, *args, **kwargs)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 6701, in map_partitions
return new_dd_object(graph, name, meta, divisions)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 7835, in new_dd_object
return da.Array(dsk, name=name, chunks=chunks, dtype=meta.dtype)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/array/core.py", line 1335, in __new__
meta = meta_from_array(meta, dtype=dtype)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/array/utils.py", line 57, in meta_from_array
x = x(shape=(0,) * (ndim or 0), dtype=dtype)
TypeError: Cannot interpret 'string[pyarrow]' as a data type
**Environment**:
- Dask version: 2012.12.0
- Python version: 3.10
- Operating System: M1 Mac
- Install method (conda, pip, source): conda
Contributor guide
Research direction
Start by running the self-contained example with the reported Dask and pandas string dtypes, then inspect dask_ml/preprocessing/label.py, especially LabelEncoder.fit and _check_array. Trace the conversion into Dask arrays shown in the traceback, and verify that fitting both string and string[pyarrow] inputs completes without a TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100