dask / dask/dask-ml

Hyperband: training for a longer time

Open
#715 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

Hyperband's rule of thumb works great when I want to sample more parameters than training epochs:

``` python
>>> X, y = make_classification(n_samples=1000)
>>> n_params = 200
>>> n_examples = len(X) * 200
>>> max_iter = n_params
>>> chunksize = n_examples // n_params
>>> chunksize
1000
```

This indicates the model should see the entire dataset every `partial_fit` call:

``` python
>>> from dask_ml.model_selection import HyperbandSearchCV
>>> search = HyperbandSearchCV(model, params, max_iter=200)
>>> search.metadata["n_models"]
143
```

How do I train for longer while still sampling (about) 200 parameters?

---

The rule of thumb won't work because it specifies a chunk size larger than the array. Right now, the only way is to sample more parameters:

``` python
>>> from dask_ml.model_selection import HyperbandSearchCV
>>> search = HyperbandSearchCV(model, params, max_iter=400) # to train for no more than 400 epochs
>>> search.metadata["n_models"]
415
```

That's more hyperparameters than I need to sample.

Contributor guide

Open the contributing guide

Research direction

Start with the HyperbandSearchCV entry point and reproduce the issue's max_iter=200 and max_iter=400 examples. Trace how max_iter affects training duration and metadata["n_models"]; done means allowing longer training while sampling about 200 parameters, with coverage for the requested behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.