dotnet / dotnet/machinelearning-modelbuilder

Azure Image CodeGen Failed (Low Disk Space)

Aperta
#832 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Dockerfile
Stelle
285
Fork
66
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

**System Information (please complete the following information):**
- Model Builder Version: Nightly from last week
- Visual Studio Version: VS 2019

**Describe the bug**
If disk space is too low, CodeGen fails. We should catch this error, and present an actionable error.

Ideally, we would have a retry button if CodeGen fails, which would restart the download of the Azure image model and rerun CodeGen. Otherwise the user has to restart the entire ML training process.

Stack:
```
at AzureML.AutoMLRunnerImages.d__27.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/RemoteAutoML/AutoMLRunnerImages.cs:line 231
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AzureImageClassificationExperiment.d__13.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AzureImageClassificationExperiment.cs:line 69
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 147
```

**To Reproduce**
After running Azure AutoML multiclass image classification, Model Builder stopped during CodeGen (or during model download). I checked free disk space and had 40MB free. _I expect the issue was a failure due to the low disk space; though could be another cause._

**Expected behavior**
I would expect a disk error message. Or a failure to write file. The current error message is rather generic and isn't actionable.

As mentioned above, ideally, we would have a retry button, which would restart the Azure model download and CodeGen. Otherwise the user has to restart the entire ML training process.

**Screenshots**

![image](https://user-images.githubusercontent.com/4080826/84603875-c6888180-ae46-11ea-88a2-3a4907e79ec3.png)

**Additional context**

The diskspace of the VS project folders for Azure AutoML image models adds up more quickly than I was expecting. Since I'm benchmarking across datasets and often multiple runs within a dataset, the size of each Model Builder created folder became important.

After compiling, 5 copies of `bestModel.onnx`, and 5 copies of `MLModel.zip` are present. In all a single VS project folder came out to 1.7GB:
![image](https://user-images.githubusercontent.com/4080826/84604541-9e4f5180-ae4b-11ea-8a8d-21c86b695ce3.png)

This is then multiplied by the number of runs/datasets I'm benchmarking, causing disk space issues for me.

End of log file:
```
2020-06-12 15:17:40.9359 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:43.5310 INFO Running AutoML pipeline sweep... (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:43.5310 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:43.5310 INFO Child run status:Completed: 1, Canceled: 1 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:46.6321 INFO Running AutoML pipeline sweep... (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:46.6321 INFO Child run status:Completed: 1, Canceled: 1 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:46.6321 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:49.0821 INFO Running AutoML pipeline sweep... (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:49.1091 INFO Child run status:Completed: 1, Canceled: 1 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:49.1091 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:51.1243 INFO Running AutoML pipeline sweep... (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:51.1243 INFO Child run status:Completed: 1, Canceled: 1 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:51.1313 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:53.2291 INFO Running AutoML pipeline sweep... (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:53.2461 INFO Child run status:Completed: 1, Canceled: 1 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:53.2581 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:55.2968 INFO Running AutoML pipeline sweep... (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:55.3038 INFO Child run status:Completed: 1, Canceled: 1 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:55.3038 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:57.3953 INFO Running AutoML pipeline sweep... (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:57.3953 INFO Child run status:Completed: 1, Canceled: 1 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:17:57.3953 INFO (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:18:01.4038 INFO Completed a training run (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:18:01.4198 INFO Downloading best model to C:\Users\jormont\AppData\Local\Temp\MLVSTools\NIHMalariaCellClassificationML\NIHMalariaCellClassificationML.Model\bestModel.onnx.. (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:18:01.4198 INFO Object reference not set to an instance of an object. (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2020-06-12 15:18:01.4478 DEBUG Object reference not set to an instance of an object.
at AzureML.AutoMLRunnerImages.d__27.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/RemoteAutoML/AutoMLRunnerImages.cs:line 231
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AzureImageClassificationExperiment.d__13.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/AzureImageClassificationExperiment.cs:line 69
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 147 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2020-06-14 13:57:11.5244 DEBUG Open Log FileC:\Users\jormont\AppData\Local\Temp\MLVSTools\logs\174e3458-b101-43b2-a0fb-622efc138c12.txt (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia da src/Microsoft.ML.ModelBuilder.AutoMLService/RemoteAutoML/AutoMLRunnerImages.cs alla riga 231, quindi traccia i chiamanti in AzureImageClassificationExperiment.cs alla riga 69 e AutoMLEngine.cs alla riga 147. Riproduci o esamina l’errore registrato relativo al download del modello e a CodeGen; il lavoro è completato quando gli errori dovuti a spazio su disco insufficiente producono un messaggio utile e viene affrontato il percorso di ripristino richiesto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
azure, csharp, machine-learning
Ambito
backend, machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.