Object Detection API: Confidence score getting lower with increase in number of training steps

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

Also, please understand that many of the models included in this repository are experimental and research-style code. If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

• What is the top-level directory of the model you are using: models/research/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Unix
• TensorFlow installed from (source or binary):source
• TensorFlow version (use command below):1.4
• Bazel version (if compiling from source):
• CUDA/cuDNN version:
• GPU model and memory:
• Exact command to reproduce:

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

I am trying to use the object detection API by TensorFlow to detect a particular pattern in a 3190X3190 image using faster_rcnn_inception_resnet_v2_atrous_coco. All my training images are of size 1140X1140. The pattern itself is of width 380 pixels and height 430 pixels. The pattern is made up of basic shapes such as rectangles and circles. Every rectangle and circle is of the same dimensions in both the inference and training images. I have 4000 training images. I notice even though I am getting reasonable results at around 5k steps with a confidence score of around 98%, the maximum confidence score drops to around 5% when the number of training steps increase to 8k. I have noticed almost the same thing if I increase the size of the training image to 1290x1290 by padding a black ring around it. However, I get good results if I use training images of size 1500x1500 even if the number of steps is 25k. I have also tried with training images of size 3190x3190 (i.e. same size as the inference image) and got detections with really low confidence score.

Is there any particular reason for this behavior? I would be interested to know as to how the size of the training image affecting the detection results. I am using version 1.4 of the object-detection API.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.