dotnet / dotnet/machinelearning-modelbuilder

Object detection locally stuck in training phase on tutorial dataset

Open
#2,887 4 comments 0 reactions 0 assignees View on GitHub
GS Tutorial
Dominant language
Dockerfile
Stars
285
Forks
66
PR merge metrics
No merged PRs in 30d

Description

**System Information (please complete the following information):**
- Model Builder or CLI Version: 17.18.2.2415501
- Visual Studio Version (if applicable): 2022 version 17.9.3

**Describe the bug**
- On which page of the tutorial did you run into an issue (URL): https://devblogs.microsoft.com/dotnet/object-detection-ml-dotnet-model-builder/ (Step 5 of topic "Get started with object detection locally")
- Clear description of the problem: Training locally gets stuck in the training stage, waited about 6 hours, tried CPU (Ryzen 5 7600X with 32GB ram) or GPU (RTX 3070) environment, nothing changes

**Logs**
```
2024-03-18 22:32:57.5620 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:57.5620 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:58.5660 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:58.5660 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:59.5760 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:59.5760 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:00.5960 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:00.5960 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:01.6030 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:01.6030 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:02.6175 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:02.6175 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:03.6262 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:03.6262 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:04.6287 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:04.6287 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:05.6395 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:05.6395 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:06.6409 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:06.6409 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:07.6492 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:07.6492 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:08.6499 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:08.6499 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:09.6540 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:09.6540 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:10.6595 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:10.6595 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
```

**To Reproduce**
Steps to reproduce the behavior: "Get started with object detection locally" https://devblogs.microsoft.com/dotnet/object-detection-ml-dotnet-model-builder/

**Expected behavior**
With my hardware spec and 40 images with one category detection train phase should take a few minutes

**Screenshots**
![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/10048426/520ea5d7-b68a-453c-924c-9a69b0ef5477)
![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/10048426/5654e88d-59ea-4a7b-97b0-1157ac10312f)

**Additional context**
Classification scenario works as expected locally with same dataset

![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/10048426/ef975ebf-166f-4c64-a030-7e784dd89ca4)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.