alteryx / alteryx/evalml

RandomSearch/GridSearch tuners: if search space exhausted for one pipeline type, entire search stops

Offen
#529 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Python
Sterne
850
Forks
96
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

**Problem**
#230 added `RandomSearch` and `GridSearch` tuners. Unlike the `SKOptTuner`, those tuners have potentially finite search spaces, and can eventually run out of parameters to suggest (particularly `GridSearch`).

If automl is run using either the `GridSearch` or `RandomSearch` tuner, and the search space for a particular pipeline (i.e. for `LogisticRegression`) is exhausted during automl, the entire search will be stopped.

**Scope and Priority**
Users would hit this if they a) used one of the new tuners and b) either ran a ton of pipelines or limited the hyperparameter space. So although this is important to fix, it's not a critical issue.

**Solution**
Under the current tuner / automl API, I think the desired behavior should be to only stop automl when the tuners for *all* pipelines have been exhausted, like so:

```
def _check_stopping_condition(self, start):
# get new pipeline and check tuner
self._next_pipeline_class = self._select_pipeline()
if all([self.tuners[class_name].is_search_space_exhausted() for class_name in self.tuners]):
return False
```

We'll also need a mechanism for eliminating exhausted pipelines from the set of considered pipelines in `AutoBase._select_pipelines`.

Long-term, I hope that #272 will encapsulate this sort of functionality inside the Tuner class, so that automl just has to check `tuner.is_search_space_exhausted()`.

@kmax12 @christopherbunn

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.