Azure / Azure/MachineLearningNotebooks

Retry pipeline and/or task on failure

Ouverte
#1,572 4 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
ADO Pipelines product-question
Langage dominant
Jupyter Notebook
Étoiles
4.4k
Forks
2.6k
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

I use the Python SDK to develop ML pipelines for Azure ML.

How do I get my PythonScriptStep tasks or the encompassing Pipeline object to simply rerun upon failure?
I reckon it's pretty common for pipelines to temporarily break upon temporary network, storage, etc. issues so a simple rerun / retry seems pretty basic for task orchestration frameworks to provide (see e.g. Apache Airflow).

I've spent a fair amount of time going over the documentation for Azure ML and I just can't figure out how to get "retry upon failure" behaviour.

The closest there is is the continue_on_step_failure pipeline / task parameter which doesn't really do what's needed.

Any advice please?

I've tried finding a solution on SO over here - a proposed solution uses external tools which just adds more overhead:

https://stackoverflow.com/questions/68647922/azure-machine-learning-pipeline-how-to-retry-upon-failure

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.