RandomSearch/GridSearch tuners: if search space exhausted for one pipeline type, entire search stops
- Lenguaje dominante
- Python
- Estrellas
- 850
- Forks
- 96
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
**Problem**
#230 added `RandomSearch` and `GridSearch` tuners. Unlike the `SKOptTuner`, those tuners have potentially finite search spaces, and can eventually run out of parameters to suggest (particularly `GridSearch`).
If automl is run using either the `GridSearch` or `RandomSearch` tuner, and the search space for a particular pipeline (i.e. for `LogisticRegression`) is exhausted during automl, the entire search will be stopped.
**Scope and Priority**
Users would hit this if they a) used one of the new tuners and b) either ran a ton of pipelines or limited the hyperparameter space. So although this is important to fix, it's not a critical issue.
**Solution**
Under the current tuner / automl API, I think the desired behavior should be to only stop automl when the tuners for *all* pipelines have been exhausted, like so:
```
def _check_stopping_condition(self, start):
# get new pipeline and check tuner
self._next_pipeline_class = self._select_pipeline()
if all([self.tuners[class_name].is_search_space_exhausted() for class_name in self.tuners]):
return False
```
We'll also need a mechanism for eliminating exhausted pipelines from the set of considered pipelines in `AutoBase._select_pipelines`.
Long-term, I hope that #272 will encapsulate this sort of functionality inside the Tuner class, so that automl just has to check `tuner.is_search_space_exhausted()`.
@kmax12 @christopherbunn
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.