Use as_completed in train_pipelines/score_pipelines
- Vorherrschende Sprache
- Python
- Sterne
- 850
- Forks
- 96
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
This issue tracks writing our own implementation of dask's `as_completed` function to pull off jobs from the queue when they are done. This will be used to consolidate the "while-loop" logic in `score_pipelines` and `train_pipelines`.
> It's not exactly the same because the thing you get back from `get_result` is not the same between training and scoring and the logic for handling exceptions is not the same as well.
>
> This might be a good spot for Max 's suggestion to use `as_completed` or something like it.
>
> If we do that, I think this will look like:
>
> ```python
> for computation in as_completed(computations):
> try:
> fitted_pipeline = computation.get_result()
> fitted_pipelines[fitted_pipeline.name] = fitted_pipeline
> except Exception as e:
> logger.error(f'Train error for {pipeline.name}: {str(e)}')
> tb = traceback.format_tb(sys.exc_info()[2])
> logger.error("Traceback:")
> logger.error("\n".join(tb))
> ```
>
> I think at that point we're not reusing the while-loop logic so the differences between the two methods might be clearer.
>
> If you're on board I'll file an issue after merge!
>
> _Originally posted by @freddyaboulton in https://github.com/alteryx/evalml/pull/1975#discussion_r605169137_
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.