dask / dask/dask-ml

LabelEncoder raises errors with string and string[pyarrow] types

Open
#953 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

Using the label encode with `string` and `string[pyarrow]` dtypes raises TypeErrors. The error is getting raised in dask.

```
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/array/utils.py", line 57, in meta_from_array
x = x(shape=(0,) * (ndim or 0), dtype=dtype)
TypeError: Cannot interpret 'string[pyarrow]' as a data type
```

...But I can't trigger this without the LabelEncoder.

**Minimal Complete Verifiable Example**:

```python
import dask.dataframe as dd
from dask_ml.preprocessing import LabelEncoder
import pandas as pd

df = pd.DataFrame({
"A": list("aaaabbbcccdddeeefffgggg")
})

ddf = dd.from_pandas(df, npartitions=3)

# works
lenc = LabelEncoder().fit(ddf["A"])

# TypeError: Cannot interpret 'string[pyarrow]' as a data type
lenc = LabelEncoder().fit(ddf["A"].astype("string[pyarrow]"))
```

**Full Traceback**:

Very similar tracebacks for `string` and `string[pyarrow]`

➜ python label_encoder_repro.py

Traceback (most recent call last):

File "/Users/paul/work/sources/dask-engineering/example-pipelines/criteo-HPO/label_encoder_repro.py", line 12, in
lenc = LabelEncoder().fit(ddf["A"].astype("string[pyarrow]"))
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask_ml/preprocessing/label.py", line 115, in fit
y = self._check_array(y)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask_ml/preprocessing/label.py", line 111, in _check_array
y = y.to_dask_array(lengths=True)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 1687, in to_dask_array
arr = self.values
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 3431, in values
return self.map_partitions(methods.values)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 874, in map_partitions
return map_partitions(func, self, *args, **kwargs)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 6701, in map_partitions
return new_dd_object(graph, name, meta, divisions)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/dataframe/core.py", line 7835, in new_dd_object
return da.Array(dsk, name=name, chunks=chunks, dtype=meta.dtype)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/array/core.py", line 1335, in __new__
meta = meta_from_array(meta, dtype=dtype)
File "/Users/paul/mambaforge/envs/ml-example/lib/python3.10/site-packages/dask/array/utils.py", line 57, in meta_from_array
x = x(shape=(0,) * (ndim or 0), dtype=dtype)
TypeError: Cannot interpret 'string[pyarrow]' as a data type

**Environment**:

- Dask version: 2012.12.0
- Python version: 3.10
- Operating System: M1 Mac
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Research direction

Start by running the self-contained example with the reported Dask and pandas string dtypes, then inspect dask_ml/preprocessing/label.py, especially LabelEncoder.fit and _check_array. Trace the conversion into Dask arrays shown in the traceback, and verify that fitting both string and string[pyarrow] inputs completes without a TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.