alteryx / alteryx/evalml

RandomSearch/GridSearch tuners: if search space exhausted for one pipeline type, entire search stops

オープン
#529 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
bug
主要言語
Python
スター
850
フォーク
96
PR マージ指標
30日以内にマージされた PR はありません

説明

**Problem**
#230 added `RandomSearch` and `GridSearch` tuners. Unlike the `SKOptTuner`, those tuners have potentially finite search spaces, and can eventually run out of parameters to suggest (particularly `GridSearch`).

If automl is run using either the `GridSearch` or `RandomSearch` tuner, and the search space for a particular pipeline (i.e. for `LogisticRegression`) is exhausted during automl, the entire search will be stopped.

**Scope and Priority**
Users would hit this if they a) used one of the new tuners and b) either ran a ton of pipelines or limited the hyperparameter space. So although this is important to fix, it's not a critical issue.

**Solution**
Under the current tuner / automl API, I think the desired behavior should be to only stop automl when the tuners for *all* pipelines have been exhausted, like so:

```
def _check_stopping_condition(self, start):
# get new pipeline and check tuner
self._next_pipeline_class = self._select_pipeline()
if all([self.tuners[class_name].is_search_space_exhausted() for class_name in self.tuners]):
return False
```

We'll also need a mechanism for eliminating exhausted pipelines from the set of considered pipelines in `AutoBase._select_pipelines`.

Long-term, I hope that #272 will encapsulate this sort of functionality inside the Tuner class, so that automl just has to check `tuner.is_search_space_exhausted()`.

@kmax12 @christopherbunn

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。