alteryx / alteryx/evalml

Use as_completed in train_pipelines/score_pipelines

Offen
#2,096 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @asniyaz Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Python
Sterne
850
Forks
96
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

This issue tracks writing our own implementation of dask's `as_completed` function to pull off jobs from the queue when they are done. This will be used to consolidate the "while-loop" logic in `score_pipelines` and `train_pipelines`.

> It's not exactly the same because the thing you get back from `get_result` is not the same between training and scoring and the logic for handling exceptions is not the same as well.
>
> This might be a good spot for Max 's suggestion to use `as_completed` or something like it.
>
> If we do that, I think this will look like:
>
> ```python
> for computation in as_completed(computations):
> try:
> fitted_pipeline = computation.get_result()
> fitted_pipelines[fitted_pipeline.name] = fitted_pipeline
> except Exception as e:
> logger.error(f'Train error for {pipeline.name}: {str(e)}')
> tb = traceback.format_tb(sys.exc_info()[2])
> logger.error("Traceback:")
> logger.error("\n".join(tb))
> ```
>
> I think at that point we're not reusing the while-loop logic so the differences between the two methods might be clearer.
>

> If you're on board I'll file an issue after merge!
>
> _Originally posted by @freddyaboulton in https://github.com/alteryx/evalml/pull/1975#discussion_r605169137_

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.