deepinsight / deepinsight/insightface
When the arcface_orch model training exceeds 2000 steps, the loss value becomes nan.
- Dominant language
- Python
- Stars
- 29.7k
- Forks
- 6.1k
- PR merge metrics
- No merged PRs in 30d
Description
Training: 2021-08-18 05:59:59,872-Speed 463.03 samples/sec Loss 44.1258 LearningRate 0.0781 Epoch: 0 Global Step: 2750 Fp16 Grad Scale: 12800 Required: 237 hours
1050 Training: 2021-08-18 06:00:43,232-Speed 461.27 samples/sec Loss 43.0147 LearningRate 0.0781 Epoch: 0 Global Step: 2800 Fp16 Grad Scale: 12800 Required: 236 hours
1051 Training: 2021-08-18 06:01:24,801-Speed 481.14 samples/sec Loss nan LearningRate 0.0781 Epoch: 0 Global Step: 2850 Fp16 Grad Scale: 0 Required: 236 hours
1052 Training: 2021-08-18 06:02:06,191-Speed 483.22 samples/sec Loss nan LearningRate 0.0781 Epoch: 0 Global Step: 2900 Fp16 Grad Scale:
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.