Tensorflow Object Detection API - SSD Continuously Increasing RAM Usage during Training

System information

• What is the top-level directory of the model you are using: models/research/object_detection/
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.10.1
• Bazel version (if compiling from source):
• CUDA/cuDNN version: CUDA 9.0 / cuDNN 7.1
• GPU model and memory: GeForce GTX 1080 Ti / 12GB
• Exact command to reproduce: python object_detection/model_main.py --pipeline_config_path=${PIPELINE_CONFIG_PATH} --model_dir=${MODEL_DIR} --alsologtostderr

Describe the problem

I use the object detection API to train different models (ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync, ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync, faster_rcnn_resnet50_coco) on my own dataset. The faster rcnn can run without problem, but the ssd models will continuously increasing RAM usage during training and finally got OOM. There is a similar problem on SO.

Source code / logs

NA