dotnet / dotnet/machinelearning

Training with object detection , Loss : Nan

Open
#7,027 10 comments 0 reactions 0 assignees View on GitHub
area-Vision untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Hello,

I'm working on object detection with Coco data ( two datasets 1855 images or 58661 images)
After some time I receive a Loss: NaN

What's the problem please knowing that I tested different datasets ?

Any suggestion please? , Thanks

System :window
Training on CPU
Batch size : 5 ~ 10
![image](https://github.com/dotnet/machinelearning-modelbuilder/assets/48053323/72387d94-e740-43cc-a02a-56240f88c060)

## Update on 2024/02/23 (from @LittleLittleCloud )

@LittleLittleCloud Get it reproduced on the following dataset using mlnet 16.18.2, CPU and batch size to 10, epoch to 1. On GPU the training is successful
- https://automlbenchmark.blob.core.windows.net/dataset/od.v1i.coco.zip

First install `mlnet-win-x64` 16.18.2

```
dotnet tool install --global mlnet-win-x64 --version 16.18.2
```

Then, kick off the training
```
mlnet object-detection --dataset /path/to/coco.json --device cpu --epoch 1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.