Evaluation in Object detection API - Hanging (or) Not working

System information

• What is the top-level directory of the model you are using: tensorflow/models/ and tensorflow/models/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Not really, except for generating tfRecords
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): binary ( I used pip3 install tensorflow-gpu, so not sure whether it's binary or source)
• TensorFlow version (use command below): 1.5.0
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 9.0.176
• GPU model and memory:
• Exact command to reproduce:

python eval.py \
   --logtostderr \
   --pipeline_config_path= training/pipeline_faster_rcnn_inception.config \
  --checkpoint_dir = training
  --eval_dir = eval

Describe the problem

This problem is similar to Evaluation in Object Detection hanging #2225

I am able to train with the object detecion API on my own dataset, which I created myself and converted them to TFrecords.

I used "faster_rcnn_inception_v2_coco_2017_11_08" model and trained it using train.py.
I used the existing ".config" file inside the model and only adjusted the paths and num_classes. The training runs like a charm and I can even see the training loss and other graphs in tensorboard, but when I start the eval.py script, it just shows this message (attached below) & nothing happens after that

INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
WARNING:tensorflow:From /home/prasanth/deeplearning/image_detection_tf_1_5/models-r1.5/research/object_detection/core/box_predictor.py:380: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead

Without viewing the mAP or loss of the eval dataset, I can't decide when to stop training and choose the checkpoint that has best eval mAP or loss.

@karmel : It would be great if you can look into this!

Source code / logs

pipeline_faster_rcnn_inception.config

• I usually run the training from "tensorflow/models/" directory

model {
  faster_rcnn {
	num_classes: 2
	image_resizer {
  	keep_aspect_ratio_resizer {
    	min_dimension: 600
    	max_dimension: 1024
  	}
	}
	feature_extractor {
  	type: "faster_rcnn_inception_v2"
  	first_stage_features_stride: 16
	}
	first_stage_anchor_generator {
  	grid_anchor_generator {
    	height_stride: 16
    	width_stride: 16
    	scales: 0.25
    	scales: 0.5
    	scales: 1.0
    	scales: 2.0
    	aspect_ratios: 0.5
    	aspect_ratios: 1.0
    	aspect_ratios: 2.0
  	}
	}
	first_stage_box_predictor_conv_hyperparams {
  	op: CONV
  	regularizer {
    	l2_regularizer {
      	weight: 0.0
    	}
  	}
  	initializer {
    	truncated_normal_initializer {
      	stddev: 0.00999999977648
    	}
  	}
	}
	first_stage_nms_score_threshold: 0.0
	first_stage_nms_iou_threshold: 0.699999988079
	first_stage_max_proposals: 300
	first_stage_localization_loss_weight: 2.0
	first_stage_objectness_loss_weight: 1.0
	initial_crop_size: 14
	maxpool_kernel_size: 2
	maxpool_stride: 2
	second_stage_box_predictor {
  	mask_rcnn_box_predictor {
    	fc_hyperparams {
      	op: FC
      	regularizer {
        	l2_regularizer {
          	weight: 0.0
        	}
      	}
      	initializer {
        	variance_scaling_initializer {
          	factor: 1.0
          	uniform: true
          	mode: FAN_AVG
        	}
      	}
    	}
    	use_dropout: false
    	dropout_keep_probability: 1.0
  	}
	}
	second_stage_post_processing {
  	batch_non_max_suppression {
    	score_threshold: 0.0
    	iou_threshold: 0.600000023842
    	max_detections_per_class: 100
    	max_total_detections: 300
  	}
  	score_converter: SOFTMAX
	}
	second_stage_localization_loss_weight: 2.0
	second_stage_classification_loss_weight: 1.0
  }
}
train_config {
  batch_size: 1
  data_augmentation_options {
	random_horizontal_flip {
	}
  }
  optimizer {
	momentum_optimizer {
  	learning_rate {
    	manual_step_learning_rate {
      	initial_learning_rate: 0.000199999994948
      	schedule {
        	step: 0
        	learning_rate: 0.000199999994948
      	}
      	schedule {
        	step: 900000
        	learning_rate: 1.99999994948e-05
      	}
      	schedule {
        	step: 1200000
        	learning_rate: 1.99999999495e-06
      	}
    	}
  	}
  	momentum_optimizer_value: 0.899999976158
	}
	use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "object_detection/faster_rcnn_inception_v2_coco_2017_11_08/model.ckpt"
  from_detection_checkpoint: true
  num_steps: 200000
}
train_input_reader {
  label_map_path: "object_detection/data/object-detection.pbtxt"
  tf_record_input_reader {
	input_path:  "object_detection/data/train.record"
  }
}
eval_config {
  num_examples: 8000
  max_evals: 10
  use_moving_averages: false
}
eval_input_reader {
  label_map_path: "object_detection/data/object-detection.pbtxt"
  shuffle: false
  num_readers: 1
  tf_record_input_reader {
	input_path:  "object_detection/data/test.record"
  }
}

eval.py

# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# 	http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

r"""Evaluation executable for detection models.

This executable is used to evaluate DetectionModels. There are two ways of
configuring the eval job.

1) A single pipeline_pb2.TrainEvalPipelineConfig file maybe specified instead.
In this mode, the --eval_training_data flag may be given to force the pipeline
to evaluate on training data instead.

Example usage:
	./eval \
    	--logtostderr \
    	--checkpoint_dir=path/to/checkpoint_dir \
    	--eval_dir=path/to/eval_dir \
    	--pipeline_config_path=pipeline_config.pbtxt

2) Three configuration files may be provided: a model_pb2.DetectionModel
configuration file to define what type of DetectionModel is being evaluated, an
input_reader_pb2.InputReader file to specify what data the model is evaluating
and an eval_pb2.EvalConfig file to configure evaluation parameters.

Example usage:
	./eval \
    	--logtostderr \
    	--checkpoint_dir=path/to/checkpoint_dir \
    	--eval_dir=path/to/eval_dir \
    	--eval_config_path=eval_config.pbtxt \
    	--model_config_path=model_config.pbtxt \
    	--input_config_path=eval_input_config.pbtxt
"""
import functools
import os
import tensorflow as tf

from object_detection import evaluator
from object_detection.builders import input_reader_builder
from object_detection.builders import model_builder
from object_detection.utils import config_util
from object_detection.utils import label_map_util


tf.logging.set_verbosity(tf.logging.INFO)

flags = tf.app.flags
flags.DEFINE_boolean('eval_training_data', False,
                 	'If training data should be evaluated for this job.')
flags.DEFINE_string('checkpoint_dir', '',
                	'Directory containing checkpoints to evaluate, typically '
                	'set to `train_dir` used in the training job.')
flags.DEFINE_string('eval_dir', '',
                	'Directory to write eval summaries to.')
flags.DEFINE_string('pipeline_config_path', '',
                	'Path to a pipeline_pb2.TrainEvalPipelineConfig config '
                	'file. If provided, other configs are ignored')
flags.DEFINE_string('eval_config_path', '',
                	'Path to an eval_pb2.EvalConfig config file.')
flags.DEFINE_string('input_config_path', '',
                	'Path to an input_reader_pb2.InputReader config file.')
flags.DEFINE_string('model_config_path', '',
                	'Path to a model_pb2.DetectionModel config file.')
flags.DEFINE_boolean('run_once', False, 'Option to only run a single pass of '
                 	'evaluation. Overrides the `max_evals` parameter in the '
                 	'provided config.')
FLAGS = flags.FLAGS


def main(unused_argv):
  assert FLAGS.checkpoint_dir, '`checkpoint_dir` is missing.'
  assert FLAGS.eval_dir, '`eval_dir` is missing.'
  tf.gfile.MakeDirs(FLAGS.eval_dir)
  if FLAGS.pipeline_config_path:
	configs = config_util.get_configs_from_pipeline_file(
    	FLAGS.pipeline_config_path)
	tf.gfile.Copy(FLAGS.pipeline_config_path,
              	os.path.join(FLAGS.eval_dir, 'pipeline.config'),
              	overwrite=True)
  else:
	configs = config_util.get_configs_from_multiple_files(
    	model_config_path=FLAGS.model_config_path,
    	eval_config_path=FLAGS.eval_config_path,
    	eval_input_config_path=FLAGS.input_config_path)
	for name, config in [('model.config', FLAGS.model_config_path),
                     	('eval.config', FLAGS.eval_config_path),
                     	('input.config', FLAGS.input_config_path)]:
  	tf.gfile.Copy(config,
                	os.path.join(FLAGS.eval_dir, name),
                	overwrite=True)

  model_config = configs['model']
  eval_config = configs['eval_config']
  if FLAGS.eval_training_data:
	input_config = configs['train_input_config']
  else:
	input_config = configs['eval_input_config']

  model_fn = functools.partial(
  	model_builder.build,
  	model_config=model_config,
  	is_training=False)

  create_input_dict_fn = functools.partial(
  	input_reader_builder.build,
  	input_config)

  label_map = label_map_util.load_labelmap(input_config.label_map_path)
  max_num_classes = max([item.id for item in label_map.item])
  categories = label_map_util.convert_label_map_to_categories(
  	label_map, max_num_classes)

  if FLAGS.run_once:
	eval_config.max_evals = 1

  evaluator.evaluate(create_input_dict_fn, model_fn, eval_config, categories,
                 	FLAGS.checkpoint_dir, FLAGS.eval_dir)


if __name__ == '__main__':
  tf.app.run()