None of the Tensorflow 2.0 models from the models zoo working on documents

I have a dataset of around 100 images from documents having tables, all the tables and signatures are annotated correctly. I used to train Faster RCNN InceptionV2 from tf1 models zoo and the accuracy was perfect. I was just trying to upgrade to tf2 models, I have been training following models:

Faster R-CNN ResNet50 V1 1024x1024
SSD ResNet50 V1 FPN 640x640 (RetinaNet50)
CenterNet Resnet101 V1 FPN 512x512
SSD MobileNet V2 FPNLite 640x640
CenterNet HourGlass104 512x512
Faster R-CNN Inception ResNet V2 640x640

I have trained around 200k steps on each of them. None of them works with the same data which works perfectly on Faster RCNN InceptionV2 from tf1 models zoo. When I try any image, the bounding boxes are not correct and their score is not more than 0.2 and in most of the cases score is less than 0.1 or 0.001. I have tried various combinations of configurations, for example:

fine_tune_checkpoint_type: "detection"
fine_tune_checkpoint_type: "classification"
fine_tune_checkpoint_type: "fine_tune" , this throws error however

I resized the training and test images both to 1024x1024, this helped increasing accuracy a little bit like from 0.01 to 0.5 but still I can't see the detection_scores in 90's like they used to be in tf1. The maximum detection score:

print('detection_scores max === ', detections['detection_scores'].numpy().max())
I can see it's 0.40230057 and usually a number like that.

P.S. I have tried another dataset on the same model, it seems to work. On the tables dataset, I can see the loss decreasing over time but the accuracy also decreases, so for example, I tried an image at 22.ckpt and the tables detected had score like 0.7 and then after training more like 100k steps when I try it on 101.ckpt, the detections have been changed and the score is 0.4 or similar.