GPU Out of memory error from commit 451906e4e82f19712455066c1b27e2a6ba71b1dd

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/model_main.py

https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/faster_rcnn_resnet101_coco.config

2. Describe the bug

I tried doing transfer learning on Faster-RCNN-resnet101-coco. Execution reaches till Step 0 and then crashes due to out of memory. This has started after below commit:
451906e4e82f19712455066c1b27e2a6ba71b1dd
All commits before this didn't give this error. Looks like some issue with tfslim.

3. Steps to reproduce

Checkout later master branch. Attempt transfer learning on Faster-RCNN-Resnet101-COCO.

4. Expected behavior

Transfer learning steps happening successfully.

5. Additional context

Out of memory error.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• Mobile device name if the issue happens on a mobile device: NA
• TensorFlow installed from (source or binary): Google Colab preinstalled.
• TensorFlow version (use command below): v1.15.2-0-g5d80e1e8e6 1.15.2
• Python version: 3.6.9
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 10.0
• GPU model and memory: GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5), 12GB

Environment:
tf_env.txt