deepinsight / deepinsight/insightface

scrfd train loss nan

Open
#1,566 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
29.8k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

训练SCRFD的时候,loss全是nan
`2021-06-05 22:37:29,092 - mmdet - INFO - Epoch [1][100/403] lr: 6.693e-04, eta: 1 day, 4:41:52, time: 0.401, data_time: 0.225, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan

2021-06-05 22:37:48,393 - mmdet - INFO - Epoch [1][200/403] lr: 1.335e-03, eta: 21:15:04, time: 0.193, data_time: 0.022, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan

2021-06-05 22:38:07,722 - mmdet - INFO - Epoch [1][300/403] lr: 2.001e-03, eta: 18:46:18, time: 0.193, data_time: 0.021, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan

2021-06-05 22:38:26,780 - mmdet - INFO - Epoch [1][400/403] lr: 2.667e-03, eta: 17:28:49, time: 0.191, data_time: 0.021, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan

2021-06-05 22:39:08,506 - mmdet - INFO - Epoch [2][100/403] lr: 3.353e-03, eta: 19:38:28, time: 0.404, data_time: 0.216, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan

2021-06-05 22:39:27,880 - mmdet - INFO - Epoch [2][200/403] lr: 4.019e-03, eta: 18:40:26, time: 0.194, data_time: 0.020, memory: 2939, loss_cls: nan, loss_bbox: nan, loss: nan

2021-06-05 22:39:47,128 - mmdet - INFO - Epoch [2][300/403] lr: 4.685e-03, eta: 17:58:03, time: 0.192, data_time: 0.020, memory: 3052, loss_cls: nan, loss_bbox: nan, loss: nan`

训练代码:CUDA_VISIBLE_DEVICES="0,1,2,3" PORT=29701 bash ./tools/dist_train.sh ./configs/scrfd/scrfd_10g.py 4
什么参数都没改,就用作者开源的,请问下大佬是啥原因呢?谢谢@nttstar

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the report with configs/scrfd/scrfd_10g.py and tools/dist_train.sh using the shown four-GPU command. Inspect the training configuration and data or loss inputs around the first logged NaN values, then compare a run with the unmodified author configuration. Done means the cause is identified and training produces finite loss values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.