deepinsight / deepinsight/insightface
scrfd train loss nan
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 29.8k
- Forks
- 6.1k
- PR merge metrics
- No merged PRs in 30d
Description
训练SCRFD的时候,loss全是nan
`2021-06-05 22:37:29,092 - mmdet - INFO - Epoch [1][100/403] lr: 6.693e-04, eta: 1 day, 4:41:52, time: 0.401, data_time: 0.225, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan
2021-06-05 22:37:48,393 - mmdet - INFO - Epoch [1][200/403] lr: 1.335e-03, eta: 21:15:04, time: 0.193, data_time: 0.022, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan
2021-06-05 22:38:07,722 - mmdet - INFO - Epoch [1][300/403] lr: 2.001e-03, eta: 18:46:18, time: 0.193, data_time: 0.021, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan
2021-06-05 22:38:26,780 - mmdet - INFO - Epoch [1][400/403] lr: 2.667e-03, eta: 17:28:49, time: 0.191, data_time: 0.021, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan
2021-06-05 22:39:08,506 - mmdet - INFO - Epoch [2][100/403] lr: 3.353e-03, eta: 19:38:28, time: 0.404, data_time: 0.216, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan
2021-06-05 22:39:27,880 - mmdet - INFO - Epoch [2][200/403] lr: 4.019e-03, eta: 18:40:26, time: 0.194, data_time: 0.020, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan
2021-06-05 22:39:47,128 - mmdet - INFO - Epoch [2][300/403] lr: 4.685e-03, eta: 17:58:03, time: 0.192, data_time: 0.020, memory: 3052, loss_cls: nan, loss_bbox: nan, loss: nan`
训练代码:CUDA_VISIBLE_DEVICES="0,1,2,3" PORT=29701 bash ./tools/dist_train.sh ./configs/scrfd/scrfd_10g.py 4
什么参数都没改,就用作者开源的,请问下大佬是啥原因呢?谢谢@nttstar
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the report with configs/scrfd/scrfd_10g.py and tools/dist_train.sh using the shown four-GPU command. Inspect the training configuration and data or loss inputs around the first logged NaN values, then compare a run with the unmodified author configuration. Done means the cause is identified and training produces finite loss values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100