deepjavalibrary / deepjavalibrary/djl
EasyTrain.fit throws "Gradient values are all zeros..." in Trainer.checkGradients
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 759
- Avg merge
- 16h 15m
- Merged PRs (30d)
- 15
Description
## Description
currently, during training, especially with small batch size (1), prediction of a batch may be equal to the label. this results in zero loss which results in above exception, which interrupts the training.
a typical example which may cause this exception is a simple MLP training the XOR function with batch size 1.
in this case the training should ignore the current batch and continue with the next one.
as i understand, there are different ways to implement this.
1. Trainer.checkGradients returns a boolean instead of throwing an exception.
2. in EasyTrain.fit add try catch around trainer.step
3. 3. in EasyTrain.trainSplit, check if loss is zero. invoke collector.backward if loss is non zero.
Will this change the current api? How?
no,
Who will benefit from this enhancement?
all.
## References
- list reference and related literature
- list known implementations
Contributor guide
Assessment
This issue has not been assessed yet.