alteryx / alteryx/evalml

RandomSearch/GridSearch tuners: if search space exhausted for one pipeline type, entire search stops

Abierto
#529 0 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
850
Forks
96
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

**Problem**
#230 added `RandomSearch` and `GridSearch` tuners. Unlike the `SKOptTuner`, those tuners have potentially finite search spaces, and can eventually run out of parameters to suggest (particularly `GridSearch`).

If automl is run using either the `GridSearch` or `RandomSearch` tuner, and the search space for a particular pipeline (i.e. for `LogisticRegression`) is exhausted during automl, the entire search will be stopped.

**Scope and Priority**
Users would hit this if they a) used one of the new tuners and b) either ran a ton of pipelines or limited the hyperparameter space. So although this is important to fix, it's not a critical issue.

**Solution**
Under the current tuner / automl API, I think the desired behavior should be to only stop automl when the tuners for *all* pipelines have been exhausted, like so:

```
def _check_stopping_condition(self, start):
# get new pipeline and check tuner
self._next_pipeline_class = self._select_pipeline()
if all([self.tuners[class_name].is_search_space_exhausted() for class_name in self.tuners]):
return False
```

We'll also need a mechanism for eliminating exhausted pipelines from the set of considered pipelines in `AutoBase._select_pipelines`.

Long-term, I hope that #272 will encapsulate this sort of functionality inside the Tuner class, so that automl just has to check `tuner.is_search_space_exhausted()`.

@kmax12 @christopherbunn

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.