Encounter Profiling failure on CUDNN engine 1: RESOURCE_EXHAUSTED: Out of memory. Was able to train the same dataset on same machine for TFLite model

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [yes] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [yes] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [ yes] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

Am using this guide to train custom object detector
https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html

2. Describe the bug

I setup my Windows 10 machine as per the guide above. I managed to run test script as per:

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2.md

with no errors (see attached testOk.txt)
testOk.txt

However, when I run the actual training with model_main_tf2.py, I ran into the above mentioned error (see outMemory.txt):

outMemory.txt

3. Steps to reproduce

Follow instructions as per

https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html

pipeline.config as attached (pipeline.config.txt). Training Batch has been reduced to 2 and this out of memory issue still occur. Machine has been restarted before running the training. System has 16Gb RAM with only 4Gb consumed before training begin.

pipeline.config.txt

4. Expected behavior

Model to be trained. As the same machine has trained a TFlite model using the same dataset, an resource issue was not expected.

5. Additional context

Step to train TFLite model as attached

tfLiteModelTraining.txt

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10

• Mobile device name if the issue happens on a mobile device:

• TensorFlow installed from (source or binary):
  pip install --ignore-installed --upgrade tensorflow==2.5.0

• TensorFlow version (use command below): v2.8.0-rc1-32-g3f878cff5b6 2.8.0

• Python version: 3.9.7

• Bazel version (if compiling from source): -

• GCC/Compiler version (if compiling from source): -

• CUDA/cuDNN version: 11.6

• GPU model and memory: Nvidia Geforce MX150, GPU memory 10Gb. System RAM 16Gb