dask / dask/dask-ml

Include Bayesian sampling in Hyperband implementation

Open
#697 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

The paper "[BOHB: Robust and Efficient Hyperparameter Optimization at Scale][1]" includes an interesting parallelization technique for Bayesian sampling in a Hyperband implementation. In Section 4.2 the describe a scheme that does the following:

1. Has a global hyperparameter space for Bayesian sampling. This hyperparameter space will be refined over time according to the Bayesian sampling principle.
2. Initializes models in a particular order:
* At first, initialize `num_workers` models. Train them as the most aggressive bracket of Hyperband specifies.
* When a model is stopped, initialize a new model with parameters sampled from the current hyperparameter space estimate. This model is from the most aggressive bracket if that bracket is not complete; otherwise it's from the next most aggressive bracket.

The number of workers will definitely influence performance: if there are infinite workers, the Bayesian sampling algorithm will not have time to run any inference on the best set of parameters. Likewise, if there's one worker Bayesian sampling can do as much inference as possible.

They show this performance:

Screen Shot 2020-07-12 at 5 34 00 PM

Similar to Dask-ML's benchmark, they start saturating between 16 and 32 workers.

[1]:http://proceedings.mlr.press/v80/falkner18a/falkner18a.pdf

Contributor guide

Open the contributing guide

Research direction

Start by reading Section 4.2 of the linked BOHB paper and then locate the existing Hyperband implementation in dask-ml. Define how Bayesian sampling should interact with worker count, bracket progression, and stopped models; done means the described sampling workflow is implemented and its performance can be compared with the cited benchmark.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.