deeplearning4j / deeplearning4j/deeplearning4j-examples

NaNs during training when running the TinyYoloHouseNumberDetection example

Open
#1,084 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.5k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

#### Issue Description

Please describe your issue, along with:
- expected behavior: the network trains normally, no NaNs are reported
- encountered behavior: training crashes after the first fit() call because of NaNs (I enabled NaN detection, see below)

#### Version Information

Please indicate relevant versions, including, if relevant:

* Deeplearning4j version: 1.0.0-beta7 and 1.0.0-M2.1
* platform information: macOS Sonoma 14.2.1

The mentioned example:
https://github.com/deeplearning4j/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/modelling/objectdetection/TinyYoloHouseNumberDetection.java

You'll need to add this as the first call in the main() method:
```java
Nd4j.getExecutioner().setProfilingConfig(ProfilerConfig.builder()
.checkForINF(true)
.checkForNAN(true)
.checkElapsedTime(true)
.checkLocality(true)
.checkWorkspaces(true)
.build());
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.