Pre-evaluation period lags and returns memory allocation errors

System information

• What is the top-level directory of the model you are using: object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS 7.6.1810, Ubuntu 18.04 (container)
• TensorFlow installed from (source or binary): pip
• TensorFlow version (use command below): 1.14.0
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 10.1 (CUDA)
• GPU model and memory: GeForce GTX 1070 Ti and 8 GB
• Exact command to reproduce:

Describe the problem

I am trying to train an instance segmentation model on a custom dataset using the implementation of Mask-RCNN in Tensorflow's Object Detection API. For reproducibility and deployment purposes, I am running an Ubuntu container inside a CentOS desktop, with both using the same CUDA version. Training periods seem to develop normally given the computational advantage of a CUDA-enabled GPU, nonetheless, a small time segment prior to the beginning of the evaluation takes nearly an hour before completing, while the system throws several OOM and memory allocation errors pertaining to the GPU's RAM. Perhaps, the exact source of the problem could be inferred more accurately by reviewing the log timestamps as provided below.

So far, I've experimented with different 'queue' values in both input reader configs, but without overcoming the problem and getting evaluation to complete in a matter of minutes like it does in other environments such as GCP's VM instances and Google Colab with comparable GPUs. It's important to note that the most sensitive hyperparameter to occasional out-of-memory or memory allocation errors during training periods (which are almost non-existent when using the Inception V2 architecture) is the min/max dimensions in image resizer, but these are rarely found and every single time the app will most likely terminate after a few (3-4) evaluations which typically take more than an hour or so.

Here is the log of a typical outcome of executing an estimator object's train_and_eval within Tensorflow's Object Detection API, on my own custom dataset

tf_memory_error2.txt

Source code / logs

Although the dataset cannot be disclosed on a public repository, I am attaching the config file which is based on a custom dataset of eighteen (18) distinct classes, where TFRecords have been verified prior to distributing TFExamples into distinct shards.

mask_rcnn_inception_v2_custom.config.txt