dask / dask/dask-ml

Dask-ml OneHotEncoder much slower than expected

Open
#548 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

Hello,

Not sure if that's more a Stackoverflow issue or not but I'm posting it here for now.

After playing a bit with Dask I was very surprised by how slow the OneHotEncoder is compared to Scitkit-learn one. Here is an example showing the issue:

```
import dask.dataframe as dd
import numpy as np
import pandas as pd
from dask.distributed import Client

import time

client_ = Client()

# Create fake data

filename = '/tmp/dask-ml-encoder.csv'
df = pd.DataFrame({
'1': np.random.randint(0, 10_000, size=10_000),
'2': np.random.randint(0, 10_000, size=10_000),
})
df.to_csv(filename, index=False)

# With Pandas + Scikit-learn

from sklearn.preprocessing import OneHotEncoder

current_time = time.time()

df = pd.read_csv(filename, index_col=False)
encoder = OneHotEncoder(categories='auto', sparse=False)
encoder.fit(df)
encoder.transform(df)

print('With Pandas + Scikit-learn', np.round(time.time() - current_time, 4))

# With Dask 1 partition

from dask_ml.preprocessing import OneHotEncoder

current_time = time.time()

ddf = dd.read_csv(filename).repartition(npartitions=1)
ddf = ddf.astype('category')
ddf = ddf.categorize()

encoder = OneHotEncoder(sparse=False)
encoder.fit(ddf)
encoder.transform(ddf).compute()

print('With Dask 1 partition', np.round(time.time() - current_time, 4))

# With Dask 100 partition

from dask_ml.preprocessing import OneHotEncoder

current_time = time.time()

ddf = dd.read_csv(filename).repartition(npartitions=100)
ddf = ddf.astype('category')
ddf = ddf.categorize()

encoder = OneHotEncoder(sparse=False)
encoder.fit(ddf)
encoder.transform(ddf).compute()

print('With Dask 100 partition', np.round(time.time() - current_time, 4))
```

(you need around 5 free GBs to run it)

Results:
```
With Pandas + Scikit-learn 0.0578
With Dask 1 partition 6.9008
With Dask 100 partition 78.4399
```

The issue is even worse with larger datasets (for example Criteo 1T where the number of columns is several hundred of thousands once it's one hot encoded). On a subsample of that dataset of around 10_000 points, I can encode it in less than a second with a home made encoder. And Dask either takes hours or fails (Memory Error).

As I am new to Dask it could totally be me doing mistakes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.