ColumnTransformer: 'DataFrame' object has no attribute 'take' with sklearn >= 1.0.0
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
```
from dask_ml.compose import ColumnTransformer as dd_column_transformer
from sklearn.compose import ColumnTransformer as sk_column_transformer
from dask_ml.preprocessing import StandardScaler as dd_standard_scaler
from sklearn.preprocessing import StandardScaler as sk_standard_scaler
import dask.dataframe as dd
import pandas as pd
df = pd.DataFrame([[0, 1], [0, 1]])
# Sklearn
sk_p = sk_column_transformer([('standard_scaler', sk_standard_scaler(), [0, 1])])
print("sk_p.fit_transform(df)")
print(sk_p.fit_transform(df))
print()
# dask
dd_p = dd_column_transformer([('standard_scaler', dd_standard_scaler(), [0, 1])])
ddf = dd.from_pandas(df, npartitions=2)
print("dd_p.fit_transform(ddf).compute()")
print(dd_p.fit_transform(ddf).compute())
```
```
sk_p.fit_transform(df)
[[0. 0.]
[0. 0.]]
dd_p.fit_transform(ddf).compute()
Traceback (most recent call last):
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/joblib/parallel.py", line 822, in dispatch_one_batch
tasks = self._ready_batches.get(block=False)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/queue.py", line 167, in get
raise Empty
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tests/local_test/dask_dataframe_no_take.py", line 20, in
print(dd_p.fit_transform(ddf).compute())
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py", line 675, in fit_transform
result = self._fit_transform(X, y, _fit_transform_one)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py", line 615, in _fit_transform
for idx, (name, trans, column, weight) in enumerate(transformers, 1)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/joblib/parallel.py", line 1043, in __call__
if self.dispatch_one_batch(iterator):
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/joblib/parallel.py", line 833, in dispatch_one_batch
islice = list(itertools.islice(iterator, big_batch_size))
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/compose/_column_transformer.py", line 615, in
for idx, (name, trans, column, weight) in enumerate(transformers, 1)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 375, in _safe_indexing
return _pandas_indexing(X, indices, indices_dtype, axis=axis)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 217, in _pandas_indexing
return X.take(key, axis=axis)
File "/opt/anaconda3/envs/mlpl/lib/python3.7/site-packages/dask/dataframe/core.py", line 4167, in __getattr__
raise AttributeError("'DataFrame' object has no attribute %r" % key)
AttributeError: 'DataFrame' object has no attribute 'take'
```
**What happened**:
`from dask_ml.compose import ColumnTransformer` doesn't support `sklearn >= 1.0.0` even though it says it supports it https://github.com/dask/dask-ml/blob/cf2410042972b944569c7914352b7c9c5132d9e5/setup.py#L19
**What you expected to happen**:
use `scikit-learn==0.24.0` with the following code works
```
from dask_ml.compose import ColumnTransformer as dd_column_transformer
from sklearn.compose import ColumnTransformer as sk_column_transformer
from sklearn.preprocessing import StandardScaler as sk_standard_scaler
import dask.dataframe as dd
import pandas as pd
df = pd.DataFrame([[0, 1], [0, 1]])
# Sklearn
sk_p = sk_column_transformer([('standard_scaler', sk_standard_scaler(), [0, 1])])
print("sk_p.fit_transform(df)")
print(sk_p.fit_transform(df))
print()
# dask
dd_p = dd_column_transformer([('standard_scaler', sk_standard_scaler(), [0, 1])]) # note here I use the sklearn standard scaler transformer
ddf = dd.from_pandas(df, npartitions=2)
print("dd_p.fit_transform(ddf).compute()")
print(dd_p.fit_transform(ddf))
```
```
sk_p.fit_transform(df)
[[0. 0.]
[0. 0.]]
dd_p.fit_transform(ddf).compute()
[[0. 0.]
[0. 0.]]
```
**Minimal Complete Verifiable Example**:
**Environment**:
- Versions: dask==2021.11.1 dask-ml==2021.11.16 pandas==1.3.4 scikit-learn==1.0.0
- Python version: 3.7.10
- Operating System: macOS Big Sur
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.