dask / dask/dask-ml

Hyperparameter optimization benchmarking

Open
#643 15 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

It'd be nice to have some benchmarks for how our different hyperparameter optimizers perform. There are a few comparisons that would be useful

1. dask_ml's drop-in replacements for GridSearchCV & RandomizedSearchCV. We're able to deconstruct Pipeline objects to avoid redundant fit calls. This benchmark would compare a `GridSearchCV(Pipeline(...))` for `dask_ml.model_selection.GridSearchCV` and `sklearn.model_selection.GridSearchCV`. We'd expect Dask-ML's to perform better the more CV splits there are and the more parameters that are explored early on in the pipeline (https://github.com/dask/dask-ml/issues/141 has some discussion).
2. Scaling of Dask's joblib backend for large problems. Internally, scikit-learn uses joblib for parallel for loops. With

```python
with joblib.parallel_backend("dask"):
...
```

The items in the for loop are executed on the Dask Cluster. There are some issues with the backend (https://github.com/joblib/joblib/issues/1020, https://github.com/joblib/joblib/issues/1025). Fixing those aren't in scope for this work, but we'd like to have benchmarks to understand the current performance and measure the speedup from fixing those.
3. General performance on large datasets with Incremental, Hyperband, etc. We can't really compare to scikit-learn here, since it doesn't handle larger-than-memory datasets. @stsievert may have some thoughts / benchmarks to share here.

cc @dankerrigan. This is more than enough work I think. If you're able to make progress on any of these (or other things you think are important) it'd be great.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.