mAP decreases as training steps increase in faster rcnn tensorflow object detection api. It is a weird way of training progress for me. what do you think the problem will be? I have used faster rccn inception v2 pretrained model to finetune it on my custom dataset. the config file is almost similar with the original version. batch size 24.

Evaluate annotation type bbox
DONE (t=4.24s).
Accumulating evaluation results...
DONE (t=0.07s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.314
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.844
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.155
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.316
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.046
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.342
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.503
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.505
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:22:41
I0831 07:22:41.378933 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:22:41
INFO:tensorflow:Saving dict for global step 3366: DetectionBoxes_Precision/mAP = 0.31436983, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.31567088, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8435796, DetectionBoxes_Precision/mAP@.75IOU = 0.15527642, DetectionBoxes_Recall/AR@1 = 0.0456, DetectionBoxes_Recall/AR@10 = 0.3421, DetectionBoxes_Recall/AR@100 = 0.5034, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.50491476, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24066836, Loss/BoxClassifierLoss/localization_loss = 0.2751926, Loss/RPNLoss/localization_loss = 0.066240005, Loss/RPNLoss/objectness_loss = 0.013297786, Loss/total_loss = 0.5953988, global_step = 3366, learning_rate = 0.001, loss = 0.5953988
I0831 07:22:41.379251 140529569548160 estimator.py:2049] Saving dict for global step 3366: DetectionBoxes_Precision/mAP = 0.31436983, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.31567088, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8435796, DetectionBoxes_Precision/mAP@.75IOU = 0.15527642, DetectionBoxes_Recall/AR@1 = 0.0456, DetectionBoxes_Recall/AR@10 = 0.3421, DetectionBoxes_Recall/AR@100 = 0.5034, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.50491476, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24066836, Loss/BoxClassifierLoss/localization_loss = 0.2751926, Loss/RPNLoss/localization_loss = 0.066240005, Loss/RPNLoss/objectness_loss = 0.013297786, Loss/total_loss = 0.5953988, global_step = 3366, learning_rate = 0.001, loss = 0.5953988
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3366: training/model.ckpt-3366
I0831 07:22:41.393022 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3366: training/model.ckpt-3366
INFO:tensorflow:global_step/sec: 0.456421
I0831 07:22:49.051671 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.456421
INFO:tensorflow:loss = 0.5314422, step = 3369 (219.095 sec)
I0831 07:22:49.052883 140529569548160 basic_session_run_hooks.py:260] loss = 0.5314422, step = 3369 (219.095 sec)
INFO:tensorflow:global_step/sec: 0.529406
I0831 07:25:57.942767 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.529406
INFO:tensorflow:loss = 0.5429549, step = 3469 (188.891 sec)
I0831 07:25:57.944226 140529569548160 basic_session_run_hooks.py:260] loss = 0.5429549, step = 3469 (188.891 sec)
INFO:tensorflow:global_step/sec: 0.526695
I0831 07:29:07.805795 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526695
INFO:tensorflow:loss = 0.55023456, step = 3569 (189.863 sec)
I0831 07:29:07.807425 140529569548160 basic_session_run_hooks.py:260] loss = 0.55023456, step = 3569 (189.863 sec)
INFO:tensorflow:Saving checkpoints for 3668 into training/model.ckpt.
I0831 07:32:14.385575 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 3668 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 07:32:17.689545 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:19.105491 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:19.118811 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 07:32:19.119150 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:20.175370 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:20.192636 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 07:32:21.930514 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T07:32:21Z
I0831 07:32:21.946928 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T07:32:21Z
INFO:tensorflow:Graph was finalized.
I0831 07:32:22.385655 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 07:32:22.386379: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.386697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 07:32:22.386823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 07:32:22.386855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 07:32:22.386881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 07:32:22.386901: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 07:32:22.386938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 07:32:22.386961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 07:32:22.386989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 07:32:22.387076: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.387378: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.387638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 07:32:22.387682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 07:32:22.387696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 07:32:22.387706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 07:32:22.387813: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.388110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.388368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-3668
I0831 07:32:22.390268 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-3668
INFO:tensorflow:Running local_init_op.
I0831 07:32:23.428871 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 07:32:23.588097 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 07:32:38.782745 140526982002432 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 07:32:38.784129 140526982002432 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 07:32:38.802798 140526982002432 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.18s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.308
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.838
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.161
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.310
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.045
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.340
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.511
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.513
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:32:43
I0831 07:32:43.164965 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:32:43
INFO:tensorflow:Saving dict for global step 3668: DetectionBoxes_Precision/mAP = 0.3079519, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3095433, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83835673, DetectionBoxes_Precision/mAP@.75IOU = 0.16109954, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3405, DetectionBoxes_Recall/AR@100 = 0.5111, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5126379, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24741209, Loss/BoxClassifierLoss/localization_loss = 0.27934107, Loss/RPNLoss/localization_loss = 0.06676239, Loss/RPNLoss/objectness_loss = 0.014064799, Loss/total_loss = 0.6075804, global_step = 3668, learning_rate = 0.001, loss = 0.6075804
I0831 07:32:43.165262 140529569548160 estimator.py:2049] Saving dict for global step 3668: DetectionBoxes_Precision/mAP = 0.3079519, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3095433, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83835673, DetectionBoxes_Precision/mAP@.75IOU = 0.16109954, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3405, DetectionBoxes_Recall/AR@100 = 0.5111, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5126379, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24741209, Loss/BoxClassifierLoss/localization_loss = 0.27934107, Loss/RPNLoss/localization_loss = 0.06676239, Loss/RPNLoss/objectness_loss = 0.014064799, Loss/total_loss = 0.6075804, global_step = 3668, learning_rate = 0.001, loss = 0.6075804
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3668: training/model.ckpt-3668
I0831 07:32:43.177099 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3668: training/model.ckpt-3668
INFO:tensorflow:global_step/sec: 0.456204
I0831 07:32:47.005799 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.456204
INFO:tensorflow:loss = 0.4690953, step = 3669 (219.200 sec)
I0831 07:32:47.006930 140529569548160 basic_session_run_hooks.py:260] loss = 0.4690953, step = 3669 (219.200 sec)
INFO:tensorflow:global_step/sec: 0.53203
I0831 07:35:54.964972 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.53203
INFO:tensorflow:loss = 0.4143501, step = 3769 (187.959 sec)
I0831 07:35:54.966166 140529569548160 basic_session_run_hooks.py:260] loss = 0.4143501, step = 3769 (187.959 sec)
INFO:tensorflow:global_step/sec: 0.526431
I0831 07:39:04.923553 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526431
INFO:tensorflow:loss = 0.49879405, step = 3869 (189.959 sec)
I0831 07:39:04.925200 140529569548160 basic_session_run_hooks.py:260] loss = 0.49879405, step = 3869 (189.959 sec)
INFO:tensorflow:Saving checkpoints for 3970 into training/model.ckpt.
I0831 07:42:15.309181 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 3970 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 07:42:18.419227 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:19.822912 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:19.835297 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 07:42:19.835624 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:21.342495 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:21.358611 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 07:42:23.143416 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T07:42:23Z
I0831 07:42:23.162491 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T07:42:23Z
INFO:tensorflow:Graph was finalized.
I0831 07:42:23.599226 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 07:42:23.599888: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.600200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 07:42:23.600302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 07:42:23.600331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 07:42:23.600356: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 07:42:23.600380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 07:42:23.600399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 07:42:23.600418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 07:42:23.600439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 07:42:23.600521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.600835: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.601074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 07:42:23.601119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 07:42:23.601132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 07:42:23.601141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 07:42:23.601241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.601519: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.601751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-3970
I0831 07:42:23.603839 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-3970
INFO:tensorflow:Running local_init_op.
I0831 07:42:24.696616 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 07:42:24.879029 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 07:42:39.873946 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 07:42:39.875130 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 07:42:39.898843 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.11s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.305
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.834
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.144
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.306
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.042
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.343
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.506
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.508
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:42:44
I0831 07:42:44.210580 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:42:44
INFO:tensorflow:Saving dict for global step 3970: DetectionBoxes_Precision/mAP = 0.30534187, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3064494, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83381045, DetectionBoxes_Precision/mAP@.75IOU = 0.14356509, DetectionBoxes_Recall/AR@1 = 0.0423, DetectionBoxes_Recall/AR@10 = 0.3434, DetectionBoxes_Recall/AR@100 = 0.5065, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5080241, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2550419, Loss/BoxClassifierLoss/localization_loss = 0.29824752, Loss/RPNLoss/localization_loss = 0.06717979, Loss/RPNLoss/objectness_loss = 0.013152282, Loss/total_loss = 0.63362145, global_step = 3970, learning_rate = 0.001, loss = 0.63362145
I0831 07:42:44.210865 140529569548160 estimator.py:2049] Saving dict for global step 3970: DetectionBoxes_Precision/mAP = 0.30534187, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3064494, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83381045, DetectionBoxes_Precision/mAP@.75IOU = 0.14356509, DetectionBoxes_Recall/AR@1 = 0.0423, DetectionBoxes_Recall/AR@10 = 0.3434, DetectionBoxes_Recall/AR@100 = 0.5065, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5080241, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2550419, Loss/BoxClassifierLoss/localization_loss = 0.29824752, Loss/RPNLoss/localization_loss = 0.06717979, Loss/RPNLoss/objectness_loss = 0.013152282, Loss/total_loss = 0.63362145, global_step = 3970, learning_rate = 0.001, loss = 0.63362145
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3970: training/model.ckpt-3970
I0831 07:42:44.222068 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3970: training/model.ckpt-3970
INFO:tensorflow:global_step/sec: 0.455995
I0831 07:42:44.223970 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.455995
INFO:tensorflow:loss = 0.5270295, step = 3969 (219.300 sec)
I0831 07:42:44.224892 140529569548160 basic_session_run_hooks.py:260] loss = 0.5270295, step = 3969 (219.300 sec)
INFO:tensorflow:global_step/sec: 0.527989
I0831 07:45:53.621862 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.527989
INFO:tensorflow:loss = 0.50037175, step = 4069 (189.399 sec)
I0831 07:45:53.623555 140529569548160 basic_session_run_hooks.py:260] loss = 0.50037175, step = 4069 (189.399 sec)
INFO:tensorflow:global_step/sec: 0.522826
I0831 07:49:04.890216 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.522826
INFO:tensorflow:loss = 0.52528, step = 4169 (191.268 sec)
I0831 07:49:04.891823 140529569548160 basic_session_run_hooks.py:260] loss = 0.52528, step = 4169 (191.268 sec)
INFO:tensorflow:global_step/sec: 0.526333
I0831 07:52:14.884055 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526333
INFO:tensorflow:loss = 0.47124624, step = 4269 (189.994 sec)
I0831 07:52:14.885528 140529569548160 basic_session_run_hooks.py:260] loss = 0.47124624, step = 4269 (189.994 sec)
INFO:tensorflow:Saving checkpoints for 4271 into training/model.ckpt.
I0831 07:52:16.755300 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 4271 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 07:52:20.224051 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:21.619853 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:21.632409 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 07:52:21.632741 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:22.708257 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:22.729282 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 07:52:24.549369 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T07:52:24Z
I0831 07:52:24.564951 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T07:52:24Z
INFO:tensorflow:Graph was finalized.
I0831 07:52:25.000237 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 07:52:25.001031: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.001356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 07:52:25.001448: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 07:52:25.001479: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 07:52:25.001503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 07:52:25.001530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 07:52:25.001554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 07:52:25.001575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 07:52:25.001597: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 07:52:25.001684: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.002077: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.002366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 07:52:25.002426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 07:52:25.002443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 07:52:25.002455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 07:52:25.002587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.002997: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.003239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-4271
I0831 07:52:25.004953 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-4271
INFO:tensorflow:Running local_init_op.
I0831 07:52:26.397088 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 07:52:26.602425 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 07:52:41.601227 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 07:52:41.602574 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 07:52:41.622398 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.10s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.285
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.805
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.135
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.286
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.045
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.330
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.495
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:52:45
I0831 07:52:45.912902 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:52:45
INFO:tensorflow:Saving dict for global step 4271: DetectionBoxes_Precision/mAP = 0.28498507, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.28630382, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8050973, DetectionBoxes_Precision/mAP@.75IOU = 0.13491808, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3295, DetectionBoxes_Recall/AR@100 = 0.4946, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.49608827, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2520462, Loss/BoxClassifierLoss/localization_loss = 0.3052786, Loss/RPNLoss/localization_loss = 0.06822211, Loss/RPNLoss/objectness_loss = 0.014657295, Loss/total_loss = 0.6402043, global_step = 4271, learning_rate = 0.001, loss = 0.6402043
I0831 07:52:45.913205 140529569548160 estimator.py:2049] Saving dict for global step 4271: DetectionBoxes_Precision/mAP = 0.28498507, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.28630382, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8050973, DetectionBoxes_Precision/mAP@.75IOU = 0.13491808, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3295, DetectionBoxes_Recall/AR@100 = 0.4946, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.49608827, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2520462, Loss/BoxClassifierLoss/localization_loss = 0.3052786, Loss/RPNLoss/localization_loss = 0.06822211, Loss/RPNLoss/objectness_loss = 0.014657295, Loss/total_loss = 0.6402043, global_step = 4271, learning_rate = 0.001, loss = 0.6402043
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4271: training/model.ckpt-4271
I0831 07:52:45.924731 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4271: training/model.ckpt-4271
INFO:tensorflow:global_step/sec: 0.460804
I0831 07:55:51.896059 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.460804
INFO:tensorflow:loss = 0.38488027, step = 4369 (217.012 sec)
I0831 07:55:51.897870 140529569548160 basic_session_run_hooks.py:260] loss = 0.38488027, step = 4369 (217.012 sec)
INFO:tensorflow:global_step/sec: 0.527887
I0831 07:59:01.330513 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.527887
INFO:tensorflow:loss = 0.39409614, step = 4469 (189.434 sec)
I0831 07:59:01.332315 140529569548160 basic_session_run_hooks.py:260] loss = 0.39409614, step = 4469 (189.434 sec)
INFO:tensorflow:global_step/sec: 0.521504
I0831 08:02:13.083614 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.521504
INFO:tensorflow:loss = 0.50167036, step = 4569 (191.753 sec)
I0831 08:02:13.085605 140529569548160 basic_session_run_hooks.py:260] loss = 0.50167036, step = 4569 (191.753 sec)
INFO:tensorflow:Saving checkpoints for 4572 into training/model.ckpt.
I0831 08:02:17.270462 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 4572 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 08:02:20.573987 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:22.028924 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:22.044717 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 08:02:22.045222 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:23.154609 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:23.171390 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 08:02:25.103013 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T08:02:25Z
I0831 08:02:25.120003 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T08:02:25Z
INFO:tensorflow:Graph was finalized.
I0831 08:02:25.581543 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 08:02:25.582306: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.582646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 08:02:25.582764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 08:02:25.582823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 08:02:25.582854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 08:02:25.582882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 08:02:25.582904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 08:02:25.582928: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 08:02:25.582964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 08:02:25.583056: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.583385: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.583651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 08:02:25.583716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 08:02:25.583729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 08:02:25.583739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 08:02:25.583853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.584140: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.584399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-4572
I0831 08:02:25.586534 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-4572
INFO:tensorflow:Running local_init_op.
I0831 08:02:26.744107 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 08:02:26.950001 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 08:02:41.757927 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 08:02:41.759518 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 08:02:41.779049 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.06s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.275
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.804
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.124
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.277
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.038
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.320
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.481
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.482
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-08:02:46
I0831 08:02:46.018646 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-08:02:46
INFO:tensorflow:Saving dict for global step 4572: DetectionBoxes_Precision/mAP = 0.27549165, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27698833, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8040639, DetectionBoxes_Precision/mAP@.75IOU = 0.12408327, DetectionBoxes_Recall/AR@1 = 0.0379, DetectionBoxes_Recall/AR@10 = 0.3203, DetectionBoxes_Recall/AR@100 = 0.4809, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.48234704, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.27474567, Loss/BoxClassifierLoss/localization_loss = 0.32303578, Loss/RPNLoss/localization_loss = 0.06927199, Loss/RPNLoss/objectness_loss = 0.0134567795, Loss/total_loss = 0.68051034, global_step = 4572, learning_rate = 0.001, loss = 0.68051034
I0831 08:02:46.018949 140529569548160 estimator.py:2049] Saving dict for global step 4572: DetectionBoxes_Precision/mAP = 0.27549165, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27698833, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8040639, DetectionBoxes_Precision/mAP@.75IOU = 0.12408327, DetectionBoxes_Recall/AR@1 = 0.0379, DetectionBoxes_Recall/AR@10 = 0.3203, DetectionBoxes_Recall/AR@100 = 0.4809, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.48234704, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.27474567, Loss/BoxClassifierLoss/localization_loss = 0.32303578, Loss/RPNLoss/localization_loss = 0.06927199, Loss/RPNLoss/objectness_loss = 0.0134567795, Loss/total_loss = 0.68051034, global_step = 4572, learning_rate = 0.001, loss = 0.68051034
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4572: training/model.ckpt-4572
I0831 08:02:46.030554 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4572: training/model.ckpt-4572
INFO:tensorflow:global_step/sec: 0.455421
I0831 08:05:52.660928 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.455421
INFO:tensorflow:loss = 0.37305856, step = 4669 (219.577 sec)
I0831 08:05:52.662541 140529569548160 basic_session_run_hooks.py:260] loss = 0.37305856, step = 4669 (219.577 sec)
INFO:tensorflow:global_step/sec: 0.52417
I0831 08:09:03.438588 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.52417
INFO:tensorflow:loss = 0.3631875, step = 4769 (190.778 sec)
I0831 08:09:03.440209 140529569548160 basic_session_run_hooks.py:260] loss = 0.3631875, step = 4769 (190.778 sec)
INFO:tensorflow:global_step/sec: 0.526801
I0831 08:12:13.263454 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526801
INFO:tensorflow:loss = 0.3741885, step = 4869 (189.825 sec)
I0831 08:12:13.265093 140529569548160 basic_session_run_hooks.py:260] loss = 0.3741885, step = 4869 (189.825 sec)
INFO:tensorflow:Saving checkpoints for 4873 into training/model.ckpt.
I0831 08:12:18.910395 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 4873 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 08:12:22.216905 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:23.580093 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:23.593119 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 08:12:23.593468 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:24.650454 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:24.665254 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 08:12:26.758405 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T08:12:26Z
I0831 08:12:26.773212 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T08:12:26Z
INFO:tensorflow:Graph was finalized.
I0831 08:12:27.221370 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 08:12:27.222244: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.222666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 08:12:27.222759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 08:12:27.222803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 08:12:27.222828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 08:12:27.222849: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 08:12:27.222877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 08:12:27.222907: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 08:12:27.222938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 08:12:27.223062: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.223472: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.223841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 08:12:27.223910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 08:12:27.223933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 08:12:27.223947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 08:12:27.224092: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.224509: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.224887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-4873
I0831 08:12:27.227638 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-4873
INFO:tensorflow:Running local_init_op.
I0831 08:12:28.571145 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 08:12:28.713408 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 08:12:42.829031 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 08:12:42.830227 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 08:12:42.848253 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=3.97s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.273
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.777
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.129
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.275
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.037
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.311
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.486
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.487
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-08:12:47
I0831 08:12:47.004221 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-08:12:47
INFO:tensorflow:Saving dict for global step 4873: DetectionBoxes_Precision/mAP = 0.27279815, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27472305, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.7765998, DetectionBoxes_Precision/mAP@.75IOU = 0.12912713, DetectionBoxes_Recall/AR@1 = 0.037, DetectionBoxes_Recall/AR@10 = 0.3107, DetectionBoxes_Recall/AR@100 = 0.486, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.4874624, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2877345, Loss/BoxClassifierLoss/localization_loss = 0.3372294, Loss/RPNLoss/localization_loss = 0.070613034, Loss/RPNLoss/objectness_loss = 0.0133400755, Loss/total_loss = 0.7089169, global_step = 4873, learning_rate = 0.001, loss = 0.7089169
I0831 08:12:47.004480 140529569548160 estimator.py:2049] Saving dict for global step 4873: DetectionBoxes_Precision/mAP = 0.27279815, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27472305, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.7765998, DetectionBoxes_Precision/mAP@.75IOU = 0.12912713, DetectionBoxes_Recall/AR@1 = 0.037, DetectionBoxes_Recall/AR@10 = 0.3107, DetectionBoxes_Recall/AR@100 = 0.486, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.4874624, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2877345, Loss/BoxClassifierLoss/localization_loss = 0.3372294, Loss/RPNLoss/localization_loss = 0.070613034, Loss/RPNLoss/objectness_loss = 0.0133400755, Loss/total_loss = 0.7089169, global_step = 4873, learning_rate = 0.001, loss = 0.7089169
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4873: training/model.ckpt-4873
I0831 08:12:47.016038 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4873: training/model.ckpt-4873
INFO:tensorflow:global_step/sec: 0.460294
I0831 08:15:50.515957 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.460294
INFO:tensorflow:loss = 0.36465797, step = 4969 (217.253 sec)
I0831 08:15:50.517859 140529569548160 basic_session_run_hooks.py:260] loss = 0.36465797, step = 4969 (217.253 sec)
INFO:tensorflow:global_step/sec: 0.528299
I0831 08:18:59.802522 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.528299
INFO:tensorflow:loss = 0.34068385, step = 5069 (189.286 sec)
I0831 08:18:59.804225 140529569548160 basic_session_run_hooks.py:260] loss = 0.34068385, step = 5069 (189.286 sec)
INFO:tensorflow:global_step/sec: 0.527435
I0831 08:22:09.399181 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.527435
INFO:tensorflow:loss = 0.3287485, step = 5169 (189.596 sec)
I0831 08:22:09.400691 140529569548160 basic_session_run_hooks.py:260] loss = 0.3287485, step = 5169 (189.596 sec)
INFO:tensorflow:Saving checkpoints for 5175 into training/model.ckpt.
I0831 08:22:18.978193 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 5175 into training/model.ckpt.
INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).
I0831 08:22:21.311619 140529569548160 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
INFO:tensorflow:global_step/sec: 0.520463
I0831 08:25:21.535818 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.520463
INFO:tensorflow:loss = 0.25103626, step = 5269 (192.136 sec)
I0831 08:25:21.537050 140529569548160 basic_session_run_hooks.py:260] loss = 0.25103626, step = 5269 (192.136 sec)
INFO:tensorflow:global_step/sec: 0.529386
I0831 08:28:30.433928 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.529386
INFO:tensorflow:loss = 0.28013855, step = 5369 (188.899 sec)
I0831 08:28:30.435732 140529569548160 basic_session_run_hooks.py:260] loss = 0.28013855, step = 5369 (188.899 sec)