object detection stuck at CUDA_ERROR_OUT_OF_MEMORY, tried every solution, not working

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

what is this????

2. Describe the bug

I was following this guide(https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html) to try the object detection feature with tensorflow gpu on my windows 10 computer，everything works fine, until i train with this line code:
python model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config

the console give me a 'CUDA_ERROR_OUT_OF_MEMORY', followed by a lot of similar errors:

the solutions i tried:
1.change my batch_size to 1, has not any effect, still oom
2.set_memory_growth to true, it has some effect, the memory increased slowly, but result in a same error, the training process still CUDA_ERROR_OUT_OF_MEMORY, oom etc.
3.i tried tf.config.experimental.set_virtual_device_configuration(gpu, [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)]), I got the same result
...

Here is a screenshot that might give some important information:

as you can see, in the screenshot, the memory usage is just 50% at that time, but the program just report the oom error, and there was not some other program compete for the memory(nvidia-msi -l)

3. Steps to reproduce

just follow the guide of this tutorial exactly
https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html

4. Expected behavior

train my data, and show the loss and accuracy

5. Additional context

some screenshot might be helpful

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): windows 10
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary): source/pip
• TensorFlow version (use command below): 2.5.0
• Python version:3.8.10
• Bazel version (if compiling from source): ?
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:11.2
• GPU model and memory: nvidia rtx3070, 8g