Very Slow inference speed of object detection models

System information

• What is the top-level directory of the model you are using:
  models/research/object_detection/

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  no

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Windows 7 64 bit

• TensorFlow installed from (source or binary):
  "native" pip

• TensorFlow version (use command below):
  Version 1.8.0

• Bazel version (if compiling from source):

• CUDA/cuDNN version:
  CUDA® Toolkit 9.0; cuDNN v7.1

• GPU model and memory:
  GForce GTX 980 (4GB)

• Exact command to reproduce:
  "object_detection_tutorial.py" from "models-master\research\object_detection"
  i added "import time" on the top and the following command to print the processing time:

  start_time = time.time()
  # Run inference
  output_dict = sess.run(tensor_dict,
                         feed_dict={image_tensor: np.expand_dims(image, 0)})
  print("--- %s seconds ---" % (time.time() - start_time))
  

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

The processing time is very high. Using the CPU i get 1.5 Second for each image. Using the GPU i get an higher time (1.7 Second). I get bad performances whit the standard model (dogs and people on the beach) and also whit a retrained model.
I have the same problem with the Cifar10 example. The performances during the train seems to be fine (7000 samples per sec) but the processing time on the eval.py code is huge (9 Seconds).
Do you have any suggestions about that? i can find a lot of benchmarks about the training time but not many about the evaluation time so i don't know how much time should i need to process a single image.

thanks,

Adriano.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.