dask / dask/dask-ml

ColumnTransformer does not work with Dask dataframes

Open
#993 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

`dask_ml.compose.ColumnTransformer` does not work with objects of types `dask_expr._collection.DataFrame` or `dask.dataframe.core.DataFrame`.

**Minimal Complete Verifiable Example**:

```python
import numpy as np
import pandas as pd
from dask_ml.compose import ColumnTransformer
from dask_ml.preprocessing import StandardScaler

import dask.dataframe as dd
from dask.distributed import Client

client = Client()

# Create a sample dataframe
df = pd.DataFrame({"A": np.random.rand(1000)})
ddf = dd.from_pandas(df, npartitions=2)
```

ColumnTransformer, specifying the columns using strings:
```python
scaler = ColumnTransformer(
transformers=[("StandardScaler", StandardScaler(), ["A"])],
remainder="passthrough",
)
scaler.fit_transform(ddf) # or scaler.fit_transform(ddf.to_legacy_dataframe())
```
Out:
```
ValueError: Specifying the columns using strings is only supported for dataframes.
```

ColumnTransformer, specifying the columns using integers:
```python
scaler = ColumnTransformer(
transformers=[("StandardScaler", StandardScaler(), [0])],
remainder="passthrough",
)
scaler.fit_transform(ddf) # or scaler.fit_transform(ddf.to_legacy_dataframe())
```
Out:
```
AttributeError: 'DataFrame' object has no attribute 'take'
```

**Anything else we need to know?**:

Pandas data frames, i.e.

```python
scaler.fit_transform(ddf.compute())
```
works as expected.

Could be related to https://github.com/dask/dask-ml/issues/962 and https://github.com/dask/dask-ml/issues/887. If this is the same issue indeed, and there are no plans to fix it in the foreseeable future, could it better to remove it from the Dask ML API?

**Environment**:

- Dask version: 2024.4.1
- Dask ML version: 2024.4.1
- Scikit-learn version: 1.4.0
- Python version: 3.10.13
- Operating System: MacOS.
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.