foolwood / foolwood/SiamMask

exit without any wrong information present!

Open
#160 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.6k
Forks
804
PR merge metrics
No merged PRs in 30d

Description

i set the "epoch = 50", but the program at sixth epoch, the program exit without any wrong information or warning information. The last information in the terminal is
" [2020-06-28 16:54:26,232-rk0-train_siammask.py#284] Epoch: [6][16850/37500] lr: 0.005000 batch_time: 1.259789 (0.319867) data_time: 0.659337 (0.132381) rpn_cls_loss: 0.063641 (0.071975) rpn_loc_loss: 0.267109 (nan) rpn_mask_loss: 0.158198 (0.168048) siammask_loss: 6.079290 (nan) mask_iou_mean: 0.609639 (0.526512) mask_iou_at_5: 0.859155 (0.620579) mask_iou_at_7: 0.140845 (0.180679)
INFO:global:Epoch: [6][16850/37500] lr: 0.005000 batch_time: 1.259789 (0.319867) data_time: 0.659337 (0.132381) rpn_cls_loss: 0.063641 (0.071975) rpn_loc_loss: 0.267109 (nan) rpn_mask_loss: 0.158198 (0.168048) siammask_loss: 6.079290 (nan) mask_iou_mean: 0.609639 (0.526512) mask_iou_at_5: 0.859155 (0.620579) mask_iou_at_7: 0.140845 (0.180679)
[2020-06-28 16:54:28,292-rk0-log_helper.py# 97] Progress: 204350 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:26 (D:H:M)

INFO:global:Progress: 204350 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:26 (D:H:M)

[2020-06-28 16:54:45,369-rk0-train_siammask.py#284] Epoch: [6][16860/37500] lr: 0.005000 batch_time: 1.414664 (0.319940) data_time: 0.691262 (0.132429) rpn_cls_loss: 0.070713 (0.071974) rpn_loc_loss: 0.221511 (nan) rpn_mask_loss: 0.186103 (0.168048) siammask_loss: 7.036219 (nan) mask_iou_mean: 0.627533 (0.526513) mask_iou_at_5: 0.948718 (0.620586) mask_iou_at_7: 0.179487 (0.180682)
INFO:global:Epoch: [6][16860/37500] lr: 0.005000 batch_time: 1.414664 (0.319940) data_time: 0.691262 (0.132429) rpn_cls_loss: 0.070713 (0.071974) rpn_loc_loss: 0.221511 (nan) rpn_mask_loss: 0.186103 (0.168048) siammask_loss: 7.036219 (nan) mask_iou_mean: 0.627533 (0.526513) mask_iou_at_5: 0.948718 (0.620586) mask_iou_at_7: 0.179487 (0.180682)
[2020-06-28 16:54:45,374-rk0-log_helper.py# 97] Progress: 204360 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:28 (D:H:M)

INFO:global:Progress: 204360 / 1875000 [10%], Speed: 0.320 s/iter, ETA 6:04:28 (D:H:M)
"
then the program exit. why did it exit without wrong information present? Could anyone answer me ?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at train_siammask.py around line 284 and log_helper.py around line 97, using the provided epoch=50 training log as the reproduction point. Investigate why training stops after the NaN loss values and whether the termination is handled or reported. Done means identifying the exit condition and documenting or correcting the failure so the run reports it or reaches the requested epoch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.