Azure / Azure/MachineLearningNotebooks

How to resume training on low priority VMs?

オープン
#1,575 コメント 7 件 リアクション 4 件 担当者 0 名 GitHub で見る
ADO product-question Training
主要言語
Jupyter Notebook
スター
4.4k
フォーク
2.6k
PR マージ指標
30日以内にマージされた PR はありません

説明

I am running an Azure ML pipeline for Machine Learning training on a low priority compute cluster. So, occasionally, the VM will be preempted and restarted at a later time. In this case, I want to resume training from where the VM was stopped by loading the last model I saved in the `outputs` directory.

This use case is also mentioned in the [docs](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-manage-optimize-cost#low-pri-vm):
> In general, we recommend using Low-Priority VMs for Batch workloads. You should also use them where interruptions are recoverable either through resubmits (for Batch Inferencing) or through restarts (**for deep learning training with checkpointing**).

However, it seems that while the saved models stored in the `outputs` directory are still shown in the Azure ML web interface after the VM restarts, my training script can not find them in that directory. Are these files not downloaded before the script is restarted? Which directory can I use instead to store these files?

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。