Include Bayesian sampling in Hyperband implementation
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
The paper "[BOHB: Robust and Efficient Hyperparameter Optimization at Scale][1]" includes an interesting parallelization technique for Bayesian sampling in a Hyperband implementation. In Section 4.2 the describe a scheme that does the following:
1. Has a global hyperparameter space for Bayesian sampling. This hyperparameter space will be refined over time according to the Bayesian sampling principle.
2. Initializes models in a particular order:
* At first, initialize `num_workers` models. Train them as the most aggressive bracket of Hyperband specifies.
* When a model is stopped, initialize a new model with parameters sampled from the current hyperparameter space estimate. This model is from the most aggressive bracket if that bracket is not complete; otherwise it's from the next most aggressive bracket.
The number of workers will definitely influence performance: if there are infinite workers, the Bayesian sampling algorithm will not have time to run any inference on the best set of parameters. Likewise, if there's one worker Bayesian sampling can do as much inference as possible.
They show this performance:

Similar to Dask-ML's benchmark, they start saturating between 16 and 32 workers.
[1]:http://proceedings.mlr.press/v80/falkner18a/falkner18a.pdf
Contributor guide
Research direction
Start by reading Section 4.2 of the linked BOHB paper and then locate the existing Hyperband implementation in dask-ml. Define how Bayesian sampling should interact with worker count, bracket progression, and stopped models; done means the described sampling workflow is implemented and its performance can be compared with the cited benchmark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100