Evaluation in Object Detection hanging

System information

• What is the top-level directory of the model you are using: tensorflow/models/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes (well, i actually just adjusted the pipeline config to fit my dataset)
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 64bit
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.2.1
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 8.0 /5.1
• GPU model and memory: GeForce GTX1060 6GB
• Exact command to reproduce: python object_detection\eval.py --logtostderr --pipeline_config_path=C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\ pipeline.config --checkpoint_dir=C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\train\ --eval_dir=C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\eval\

Describe the problem

I am able to train with the object detecion API on my own dataset, which I created using the create_pascal_tf_record.py script (I adjusted it a bit, but mainly the paths). I also checked the generated TFRecord files with the Tensorflow Testing module and verified, that the reconstructed images are similar to the original ones.

I use the existing faster_r-cnn_resnet101_voc07.config file and only adjusted the paths and num_classes. The training runs like a charm, but when I start the eval.py script, it hangs with the message "INFO:tensorflow:Restoring parameters from C:\Users\robin\PycharmProjects\test\my-net\models\faster_r-cnn_resnet101\train\model.ckpt-123805" (see full log below).

After this I have to CTRL+C

However, I can see some output in Tensorboard, but only one value after each CTRL+C for mAP but nothing else in the other diagrams etc.

As mentioned by others having the same issue, running the evaluation and training parallel doesn't work for me, and I can't even imagine that it should be done this way. When I try it, my cuda crashed because the GPU runs out of memory.

Btw I also tried the whole evaluation process on the Oxford-IIIT Pet Dataset and am facing the same issue.

Source code / logs

The whole log after I hit CTRL+C (the part where it hangs is bold):

C:\Users\robin\models>python object_detection\eval.py --logtostderr --pipeline_config_path=C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\pipeline.config --checkpoint_dir=C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\train\ --eval_dir=C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\eval
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
2017-08-16 07:40:03.000943: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.001072: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.001933: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.002044: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.002153: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.002263: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.002357: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.002451: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-08-16 07:40:03.313527: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:940] Found device 0 with properties:
name: GeForce GTX 1060 6GB
major: 6 minor: 1 memoryClockRate (GHz) 1.7085
pciBusID 0000:01:00.0
Total memory: 6.00GiB
Free memory: 5.01GiB
2017-08-16 07:40:03.313690: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:961] DMA: 0
2017-08-16 07:40:03.314894: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0: Y
2017-08-16 07:40:03.314995: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)
INFO:tensorflow:Restoring parameters from C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\train\model.ckpt-123805
INFO:tensorflow:Restoring parameters from C:\Users\robin\PycharmProjects\test\my_net\models\faster_r-cnn_resnet101\train\model.ckpt-123805
Traceback (most recent call last):
File "object_detection\eval.py", line 161, in
tf.app.run()
File "C:\Users\robin\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection\eval.py", line 157, in main
FLAGS.checkpoint_dir, FLAGS.eval_dir)
File "C:\Users\robin\models\object_detection\evaluator.py", line 211, in evaluate
save_graph_dir=(eval_dir if eval_config.save_graph else ''))
File "C:\Users\robin\models\object_detection\eval_util.py", line 524, in repeated_checkpoint_run
time.sleep(time_to_next_eval)
KeyboardInterrupt