dotnet / dotnet/machinelearning-modelbuilder

Object detection locally stuck in training phase on tutorial dataset

Aperta
#2,887 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
GS Tutorial
Lingua principale
Dockerfile
Stelle
285
Fork
66
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

**System Information (please complete the following information):**
- Model Builder or CLI Version: 17.18.2.2415501
- Visual Studio Version (if applicable): 2022 version 17.9.3

**Describe the bug**
- On which page of the tutorial did you run into an issue (URL): https://devblogs.microsoft.com/dotnet/object-detection-ml-dotnet-model-builder/ (Step 5 of topic "Get started with object detection locally")
- Clear description of the problem: Training locally gets stuck in the training stage, waited about 6 hours, tried CPU (Ryzen 5 7600X with 32GB ram) or GPU (RTX 3070) environment, nothing changes

**Logs**
```
2024-03-18 22:32:57.5620 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:57.5620 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:58.5660 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:58.5660 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:59.5760 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:59.5760 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:00.5960 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:00.5960 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:01.6030 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:01.6030 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:02.6175 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:02.6175 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:03.6262 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:03.6262 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:04.6287 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:04.6287 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:05.6395 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:05.6395 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:06.6409 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:06.6409 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:07.6492 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:07.6492 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:08.6499 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:08.6499 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:09.6540 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:09.6540 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:10.6595 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:10.6595 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
```

**To Reproduce**
Steps to reproduce the behavior: "Get started with object detection locally" https://devblogs.microsoft.com/dotnet/object-detection-ml-dotnet-model-builder/

**Expected behavior**
With my hardware spec and 40 images with one category detection train phase should take a few minutes

**Screenshots**
![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/10048426/520ea5d7-b68a-453c-924c-9a69b0ef5477)
![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/10048426/5654e88d-59ea-4a7b-97b0-1157ac10312f)

**Additional context**
Classification scenario works as expected locally with same dataset

![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/10048426/ef975ebf-166f-4c64-a030-7e784dd89ca4)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia dal passaggio 5 del tutorial collegato “Get started with object detection locally” e riproduci la fase di training usando la versione di Model Builder e il dataset indicati. Traccia gli aggiornamenti ripetuti dei log TrainViewModel_159 RemainingSeconds e RemainingTime, quindi confronta le esecuzioni su CPU e GPU. Il lavoro è completato quando il dataset del tutorial completa il training nei pochi minuti previsti oppure il fallimento viene segnalato con una causa chiara.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
machine-learning
Ambito
machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.