ColumnTransformer on dask objects
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
Currently `sklearn.compose.ColumnTransformer` doesn't play well with dask.
1. It converts dask arrays to numpy arrays
2. It doesn't work with dask DataFrame objects
```python
In [7]: import numpy as np
...: import pandas as pd
...: import dask.dataframe as dd
...: import dask.array as da
...: from sklearn.compose import ColumnTransformer
...:
...: from sklearn.compose import ColumnTransformer
...: from dask_ml.preprocessing import RobustScaler
...:
...: ct = ColumnTransformer(
...: [("norm1", RobustScaler(), [0, 1]),
...: ("norm2", RobustScaler(), slice(2, 4))])
...:
In [8]: X = da.random.uniform(size=(10, 4), chunks=5)
...: ct.fit_transform(X)
...:
...:
Out[8]:
array([[-0.3596402 , 0.83962405, -0.40323198, 0.87204337],
[-0.95573526, -0.40216418, -0.46303583, -0.38374218],
[ 0. , -0.60132357, 0.05485146, 0.84850914],
[ 2.98179874, -1.17270669, 0.24980733, -0.10651099],
[-1.42798348, -0.48104131, -0.58646721, -0.65340624],
[ 1.04150596, 0.51895869, -0.17088259, -0.29576206],
[-0.27930786, -0.14584278, 0.59676802, 0. ],
[ 0.6403598 , 0.4256941 , -0.46941694, 0.9427417 ],
[-0.41214625, 0.82684762, 1.04350174, -0.15149086],
[ 0.29344966, 0. , 0. , 0.67583445]])
In [10]: X = dd.from_array(X)
In [11]: ct.fit_transform(X)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in ()
----> 1 ct.fit_transform(X)
~/sandbox/scikit-learn/sklearn/compose/_column_transformer.py in fit_transform(self, X, y)
383 """
384 self._validate_transformers()
--> 385 self._validate_remainder(X)
386
387 result = self._fit_transform(X, y, _fit_transform_one)
~/sandbox/scikit-learn/sklearn/compose/_column_transformer.py in _validate_remainder(self, X)
227 "'passthrough'. {0:r} was passed instead")
228
--> 229 n_columns = X.shape[1]
230
231 if self.remainder == 'passthrough':
~/sandbox/dask/dask/dataframe/core.py in __getattr__(self, key)
2417 return new_dd_object(merge(self.dask, dsk), name,
2418 meta, self.divisions)
-> 2419 raise AttributeError("'DataFrame' object has no attribute %r" % key)
2420
2421 def __dir__(self):
AttributeError: 'DataFrame' object has no attribute 'shape'
```
Ideally `Out[8]` would be a dask array
Contributor guide
Assessment
This issue has not been assessed yet.