Dask-ml OneHotEncoder much slower than expected
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
Not sure if that's more a Stackoverflow issue or not but I'm posting it here for now.
After playing a bit with Dask I was very surprised by how slow the OneHotEncoder is compared to Scitkit-learn one. Here is an example showing the issue:
```
import dask.dataframe as dd
import numpy as np
import pandas as pd
from dask.distributed import Client
import time
client_ = Client()
# Create fake data
filename = '/tmp/dask-ml-encoder.csv'
df = pd.DataFrame({
'1': np.random.randint(0, 10_000, size=10_000),
'2': np.random.randint(0, 10_000, size=10_000),
})
df.to_csv(filename, index=False)
# With Pandas + Scikit-learn
from sklearn.preprocessing import OneHotEncoder
current_time = time.time()
df = pd.read_csv(filename, index_col=False)
encoder = OneHotEncoder(categories='auto', sparse=False)
encoder.fit(df)
encoder.transform(df)
print('With Pandas + Scikit-learn', np.round(time.time() - current_time, 4))
# With Dask 1 partition
from dask_ml.preprocessing import OneHotEncoder
current_time = time.time()
ddf = dd.read_csv(filename).repartition(npartitions=1)
ddf = ddf.astype('category')
ddf = ddf.categorize()
encoder = OneHotEncoder(sparse=False)
encoder.fit(ddf)
encoder.transform(ddf).compute()
print('With Dask 1 partition', np.round(time.time() - current_time, 4))
# With Dask 100 partition
from dask_ml.preprocessing import OneHotEncoder
current_time = time.time()
ddf = dd.read_csv(filename).repartition(npartitions=100)
ddf = ddf.astype('category')
ddf = ddf.categorize()
encoder = OneHotEncoder(sparse=False)
encoder.fit(ddf)
encoder.transform(ddf).compute()
print('With Dask 100 partition', np.round(time.time() - current_time, 4))
```
(you need around 5 free GBs to run it)
Results:
```
With Pandas + Scikit-learn 0.0578
With Dask 1 partition 6.9008
With Dask 100 partition 78.4399
```
The issue is even worse with larger datasets (for example Criteo 1T where the number of columns is several hundred of thousands once it's one hot encoded). On a subsample of that dataset of around 10_000 points, I can encode it in less than a second with a home made encoder. And Dask either takes hours or fails (Memory Error).
As I am new to Dask it could totally be me doing mistakes.
Contributor guide
Assessment
This issue has not been assessed yet.