deepjavalibrary / deepjavalibrary/djl

EasyTrain.fit throws "Gradient values are all zeros..." in Trainer.checkGradients

Open
#3,696 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
4.9k
Forks
759
Avg merge
16h 15m
Merged PRs (30d)
15

Description

## Description

currently, during training, especially with small batch size (1), prediction of a batch may be equal to the label. this results in zero loss which results in above exception, which interrupts the training.

a typical example which may cause this exception is a simple MLP training the XOR function with batch size 1.

in this case the training should ignore the current batch and continue with the next one.

as i understand, there are different ways to implement this.

1. Trainer.checkGradients returns a boolean instead of throwing an exception.
2. in EasyTrain.fit add try catch around trainer.step
3. 3. in EasyTrain.trainSplit, check if loss is zero. invoke collector.backward if loss is non zero.

Will this change the current api? How?

no,

Who will benefit from this enhancement?

all.

## References

- list reference and related literature
- list known implementations

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.