alteryx / alteryx/evalml

Use as_completed in train_pipelines/score_pipelines

Aperta
#2,096 0 commenti 0 reazioni 1 assegnatario Rivendicata da @asniyaz Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
850
Fork
96
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

This issue tracks writing our own implementation of dask's `as_completed` function to pull off jobs from the queue when they are done. This will be used to consolidate the "while-loop" logic in `score_pipelines` and `train_pipelines`.

> It's not exactly the same because the thing you get back from `get_result` is not the same between training and scoring and the logic for handling exceptions is not the same as well.
>
> This might be a good spot for Max 's suggestion to use `as_completed` or something like it.
>
> If we do that, I think this will look like:
>
> ```python
> for computation in as_completed(computations):
> try:
> fitted_pipeline = computation.get_result()
> fitted_pipelines[fitted_pipeline.name] = fitted_pipeline
> except Exception as e:
> logger.error(f'Train error for {pipeline.name}: {str(e)}')
> tb = traceback.format_tb(sys.exc_info()[2])
> logger.error("Traceback:")
> logger.error("\n".join(tb))
> ```
>
> I think at that point we're not reusing the while-loop logic so the differences between the two methods might be clearer.
>

> If you're on board I'll file an issue after merge!
>
> _Originally posted by @freddyaboulton in https://github.com/alteryx/evalml/pull/1975#discussion_r605169137_

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.