trying to train imported model not loading

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ Y] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [ Y] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [ Y] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py

2. Describe the bug

so i'm trying to train a pre-trained imprted model (SSD ResNet50 V1 FPN 640x640),
using this tutorial: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html
after setting the pipeline running this commend: python model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config
this is the output:

(venv) ruper@ubuntu:~/University/FinalProject/myModel$ python3 ../models/research/object_detection/model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config
2020-11-18 14:13:56.821321: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-18 14:13:58.092171: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-11-18 14:13:58.132188: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.132593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:02:00.0 name: GeForce MX250 computeCapability: 6.1
coreClock: 1.582GHz coreCount: 3 deviceMemorySize: 1.96GiB deviceMemoryBandwidth: 44.76GiB/s
2020-11-18 14:13:58.132624: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-18 14:13:58.134297: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-18 14:13:58.135787: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-18 14:13:58.136066: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-18 14:13:58.137803: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-18 14:13:58.138776: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-18 14:13:58.142432: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-18 14:13:58.142593: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.142993: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.143285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-18 14:13:58.143590: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-18 14:13:58.150721: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 1999965000 Hz
2020-11-18 14:13:58.151092: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x60a1140 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-18 14:13:58.151112: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-11-18 14:13:58.210402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.210713: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x61396a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-11-18 14:13:58.210726: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce MX250, Compute Capability 6.1
2020-11-18 14:13:58.210875: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.211078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:02:00.0 name: GeForce MX250 computeCapability: 6.1
coreClock: 1.582GHz coreCount: 3 deviceMemorySize: 1.96GiB deviceMemoryBandwidth: 44.76GiB/s
2020-11-18 14:13:58.211097: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-18 14:13:58.211120: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-11-18 14:13:58.211134: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-11-18 14:13:58.211146: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-11-18 14:13:58.211158: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-11-18 14:13:58.211170: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-11-18 14:13:58.211183: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-11-18 14:13:58.211223: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.211436: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.211617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-11-18 14:13:58.211637: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-11-18 14:13:58.648784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-18 14:13:58.648809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-11-18 14:13:58.648818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-11-18 14:13:58.648975: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.649229: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-18 14:13:58.649429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1045 MB memory) -> physical GPU (device: 0, name: GeForce MX250, pci bus id: 0000:02:00.0, compute capability: 6.1)
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I1118 14:13:58.651196 139877428746048 mirrored_strategy.py:341] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: None
I1118 14:13:58.653778 139877428746048 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I1118 14:13:58.653855 139877428746048 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Reading unweighted datasets: ['annotations/train.record']
I1118 14:13:58.695269 139877428746048 dataset_builder.py:148] Reading unweighted datasets: ['annotations/train.record']
INFO:tensorflow:Reading record datasets for input file: ['annotations/train.record']
I1118 14:13:58.695656 139877428746048 dataset_builder.py:77] Reading record datasets for input file: ['annotations/train.record']
INFO:tensorflow:Number of filenames to read: 1
I1118 14:13:58.695724 139877428746048 dataset_builder.py:78] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W1118 14:13:58.695777 139877428746048 dataset_builder.py:85] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /home/ruper/venv/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:99: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
W1118 14:13:58.696982 139877428746048 deprecation.py:317] From /home/ruper/venv/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:99: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
WARNING:tensorflow:From /home/ruper/venv/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:221: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W1118 14:13:58.716647 139877428746048 deprecation.py:317] From /home/ruper/venv/lib/python3.8/site-packages/object_detection/builders/dataset_builder.py:221: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From /home/ruper/venv/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W1118 14:14:03.202094 139877428746048 deprecation.py:317] From /home/ruper/venv/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From /home/ruper/venv/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W1118 14:14:05.154023 139877428746048 deprecation.py:317] From /home/ruper/venv/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:201: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From /home/ruper/venv/lib/python3.8/site-packages/object_detection/inputs.py:262: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W1118 14:14:06.276827 139877428746048 deprecation.py:317] From /home/ruper/venv/lib/python3.8/site-packages/object_detection/inputs.py:262: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.

from what I saw search for an answer it should start load checkpoint of the model (correct me if i'm wrong)
it's stuck doing nothing for 3 hours and nothing happen so my guess is something not working but i can't figure out what.
I don't think it's doing something because after running it and it's got to this point i could see the gpu usage going up and after this point it's went back down and looks like the gpu isn't doing anything heavy (same usage as watching a video or something).

3. Steps to reproduce

I don't know

4. Expected behavior

expected to load checkpoint and to start training the model

5. Additional context

on the system monitor it's showing using about 40% cpu about 500m memory
on nvidia-smi: 0 N/A N/A 8754 C python3 1189MiB

6. System information

• linux ubuntu 20.04
• TensorFlow installed from pip
• TensorFlow version: 2.3.1
• Python version: 3.8.5
• CUDA/cuDNN version: 10.1
• GPU model and memory: Nvidia mx250 2GB
• CPU (idk if needed): i7 8565u
• RAM: 16GB