Same models trained on ssd mobilenet v2 fpn lite 640*640 have different FPS.

I have a 1650ti gpu 4gb and i trained an object detection model for detection of one class and then trained another model with 38 classes. The model with 1 class is giving 30 FPS but the model with 38 classes is giving 0.03 FPS. I have exported the saved model into a frozen graph and have done the inference from the checkpoint as mentioned here::
https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/auto_examples/plot_object_detection_checkpoint.html#sphx-glr-auto-examples-plot-object-detection-checkpoint-py
for both the models but one is performing fast and another one is dead slow.I am not able to figure out where i am going wrong.
I have tried out to retrain by following the tutorial again but the results are same. The accuracy is good but only fps is hampered.
Please help me out as soon as possible.

I even tried out other ssd models for multiple classes but the problem remains the same but single class models achieve the desired fps and accuracy.

Also, does increasing number of classes decreases the fps? I want to know. I have tried out all optimization techniques but all in vain.

Please let me know how to rectify this issue as soon as possible.