alteryx / alteryx/evalml

RandomSearch/GridSearch tuners: if search space exhausted for one pipeline type, entire search stops

Ouverte
#529 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
bug
Langage dominant
Python
Étoiles
850
Forks
96
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

**Problem**
#230 added `RandomSearch` and `GridSearch` tuners. Unlike the `SKOptTuner`, those tuners have potentially finite search spaces, and can eventually run out of parameters to suggest (particularly `GridSearch`).

If automl is run using either the `GridSearch` or `RandomSearch` tuner, and the search space for a particular pipeline (i.e. for `LogisticRegression`) is exhausted during automl, the entire search will be stopped.

**Scope and Priority**
Users would hit this if they a) used one of the new tuners and b) either ran a ton of pipelines or limited the hyperparameter space. So although this is important to fix, it's not a critical issue.

**Solution**
Under the current tuner / automl API, I think the desired behavior should be to only stop automl when the tuners for *all* pipelines have been exhausted, like so:

```
def _check_stopping_condition(self, start):
# get new pipeline and check tuner
self._next_pipeline_class = self._select_pipeline()
if all([self.tuners[class_name].is_search_space_exhausted() for class_name in self.tuners]):
return False
```

We'll also need a mechanism for eliminating exhausted pipelines from the set of considered pipelines in `AutoBase._select_pipelines`.

Long-term, I hope that #272 will encapsulate this sort of functionality inside the Tuner class, so that automl just has to check `tuner.is_search_space_exhausted()`.

@kmax12 @christopherbunn

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.