dask / dask/dask-ml

ColumnTransformer: 'DataFrame' object has no attribute 'take' with sklearn >= 1.0.0

Open
#887 10 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

```
from dask_ml.compose import ColumnTransformer as dd_column_transformer
from sklearn.compose import ColumnTransformer as sk_column_transformer
from dask_ml.preprocessing import StandardScaler as dd_standard_scaler
from sklearn.preprocessing import StandardScaler as sk_standard_scaler
import dask.dataframe as dd
import pandas as pd

df = pd.DataFrame([[0, 1], [0, 1]])

# Sklearn
sk_p = sk_column_transformer([('standard_scaler', sk_standard_scaler(), [0, 1])])
print("sk_p.fit_transform(df)")
print(sk_p.fit_transform(df))
print()

# dask
dd_p = dd_column_transformer([('standard_scaler', dd_standard_scaler(), [0, 1])])
ddf = dd.from_pandas(df, npartitions=2)
print("dd_p.fit_transform(ddf).compute()")
print(dd_p.fit_transform(ddf).compute())
```

```
sk_p.fit_transform(df)
[[0. 0.]
[0. 0.]]

dd_p.fit_transform(ddf).compute()
Traceback (most recent call last):
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/joblib/parallel.py", line 822, in dispatch_one_batch
tasks = self._ready_batches.get(block=False)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/queue.py", line 167, in get
raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "tests/local_test/dask_dataframe_no_take.py", line 20, in
print(dd_p.fit_transform(ddf).compute())
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py", line 675, in fit_transform
result = self._fit_transform(X, y, _fit_transform_one)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py", line 615, in _fit_transform
for idx, (name, trans, column, weight) in enumerate(transformers, 1)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/joblib/parallel.py", line 1043, in __call__
if self.dispatch_one_batch(iterator):
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/joblib/parallel.py", line 833, in dispatch_one_batch
islice = list(itertools.islice(iterator, big_batch_size))
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py", line 615, in
for idx, (name, trans, column, weight) in enumerate(transformers, 1)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 375, in _safe_indexing
return _pandas_indexing(X, indices, indices_dtype, axis=axis)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 217, in _pandas_indexing
return X.take(key, axis=axis)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/dask/dataframe/core.py", line 4167, in __getattr__
raise AttributeError("'DataFrame' object has no attribute %r" % key)
AttributeError: 'DataFrame' object has no attribute 'take'
```

**What happened**:
`from dask_ml.compose import ColumnTransformer` doesn't support `sklearn >= 1.0.0` even though it says it supports it https://github.com/dask/dask-ml/blob/cf2410042972b944569c7914352b7c9c5132d9e5/setup.py#L19

**What you expected to happen**:
use `scikit-learn==0.24.0` with the following code works
```
from dask_ml.compose import ColumnTransformer as dd_column_transformer
from sklearn.compose import ColumnTransformer as sk_column_transformer
from sklearn.preprocessing import StandardScaler as sk_standard_scaler
import dask.dataframe as dd
import pandas as pd

df = pd.DataFrame([[0, 1], [0, 1]])

# Sklearn
sk_p = sk_column_transformer([('standard_scaler', sk_standard_scaler(), [0, 1])])
print("sk_p.fit_transform(df)")
print(sk_p.fit_transform(df))
print()

# dask
dd_p = dd_column_transformer([('standard_scaler', sk_standard_scaler(), [0, 1])]) # note here I use the sklearn standard scaler transformer
ddf = dd.from_pandas(df, npartitions=2)
print("dd_p.fit_transform(ddf).compute()")
print(dd_p.fit_transform(ddf))
```
```
sk_p.fit_transform(df)
[[0. 0.]
[0. 0.]]

dd_p.fit_transform(ddf).compute()
[[0. 0.]
[0. 0.]]
```

**Minimal Complete Verifiable Example**:

**Environment**:

- Versions: dask==2021.11.1 dask-ml==2021.11.16 pandas==1.3.4 scikit-learn==1.0.0
- Python version: 3.7.10
- Operating System: macOS Big Sur
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.