exit without any wrong information present!
- Dominant language
- Python
- Stars
- 3.6k
- Forks
- 804
- PR merge metrics
- No merged PRs in 30d
Description
i set the "epoch = 50", but the program at sixth epoch, the program exit without any wrong information or warning information. The last information in the terminal is
" [2020-06-28 16:54:26,232-rk0-train_siammask.py#284] Epoch: [6][16850/37500] lr: 0.005000 batch_time: 1.259789 (0.319867) data_time: 0.659337 (0.132381) rpn_cls_loss: 0.063641 (0.071975) rpn_loc_loss: 0.267109 (nan) rpn_mask_loss: 0.158198 (0.168048) siammask_loss: 6.079290 (nan) mask_iou_mean: 0.609639 (0.526512) mask_iou_at_5: 0.859155 (0.620579) mask_iou_at_7: 0.140845 (0.180679)
INFO:global:Epoch: [6][16850/37500] lr: 0.005000 batch_time: 1.259789 (0.319867) data_time: 0.659337 (0.132381) rpn_cls_loss: 0.063641 (0.071975) rpn_loc_loss: 0.267109 (nan) rpn_mask_loss: 0.158198 (0.168048) siammask_loss: 6.079290 (nan) mask_iou_mean: 0.609639 (0.526512) mask_iou_at_5: 0.859155 (0.620579) mask_iou_at_7: 0.140845 (0.180679)
[2020-06-28 16:54:28,292-rk0-log_helper.py# 97] Progress: 204350 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:26 (D:H:M)
INFO:global:Progress: 204350 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:26 (D:H:M)
[2020-06-28 16:54:45,369-rk0-train_siammask.py#284] Epoch: [6][16860/37500] lr: 0.005000 batch_time: 1.414664 (0.319940) data_time: 0.691262 (0.132429) rpn_cls_loss: 0.070713 (0.071974) rpn_loc_loss: 0.221511 (nan) rpn_mask_loss: 0.186103 (0.168048) siammask_loss: 7.036219 (nan) mask_iou_mean: 0.627533 (0.526513) mask_iou_at_5: 0.948718 (0.620586) mask_iou_at_7: 0.179487 (0.180682)
INFO:global:Epoch: [6][16860/37500] lr: 0.005000 batch_time: 1.414664 (0.319940) data_time: 0.691262 (0.132429) rpn_cls_loss: 0.070713 (0.071974) rpn_loc_loss: 0.221511 (nan) rpn_mask_loss: 0.186103 (0.168048) siammask_loss: 7.036219 (nan) mask_iou_mean: 0.627533 (0.526513) mask_iou_at_5: 0.948718 (0.620586) mask_iou_at_7: 0.179487 (0.180682)
[2020-06-28 16:54:45,374-rk0-log_helper.py# 97] Progress: 204360 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:28 (D:H:M)
INFO:global:Progress: 204360 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:28 (D:H:M)
"
then the program exit. why did it exit without wrong information present? Could anyone answer me ?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at train_siammask.py around line 284 and log_helper.py around line 97, using the provided epoch=50 training log as the reproduction point. Investigate why training stops after the NaN loss values and whether the termination is handled or reported. Done means identifying the exit condition and documenting or correcting the failure so the run reports it or reaches the requested epoch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100