Big computational overhead for inference run?

Dear fellow developers,

I have been trying to run Deeplab v3+ with real time inference on a video, so the actual runtime for each inference is really critical to me. If I wanted to maintain 30fps, it means each inference call should take less than 30ms. However, when I tried 3 different models with much different compute, the runtime are slow and unexpected:

I went with the mobilenetv2 frozen inference graph from the official site.

Below are the three model I tried out and the code below is modified from the official Colaboratory sample by adding a simple time.time() before and after sess.run(...) to see how much time it takes per inference call.
name of model : multiply-adds
mobilenetv2_dm05_coco_voc_trainaug : 0.88B
mobilenetv2_coco_voc_trainval : 152.59B
xception65_coco_voc_trainval : 3055.35B

However I found on my i7 machine the runtimes for inferencing one image are:

mobilenetv2_dm05_coco_voc_trainaug : 170ms
mobilenetv2_coco_voc_trainval : 300ms
xception65_coco_voc_trainval : 2.5s

I'm especially confused by the first 2 models, as their multiply-adds count differ by 100 times but why the first model still takes similar amount of time for the inference_call? I was hoping it could be ~ 100 times faster so be suitable for the 30fps realtime segmentation. Does this mean there is some big overhead in the inference call? If yes, are there ways to shave them off?

Could you please advise? Thank you so much!

Best,
Bo

`
def run(self, image):
"""Runs inference on a single image.

Args:
  image: A PIL.Image object, raw input image.

Returns:
  resized_image: RGB image resized from original input image.
  seg_map: Segmentation map of `resized_image`.
"""
width, height = image.size
resize_ratio = 1.0 * self.INPUT_SIZE / max(width, height)
target_size = (int(resize_ratio * width), int(resize_ratio * height))
resized_image = image.convert('RGB').resize(target_size, Image.ANTIALIAS)

oldtime = time.time()

batch_seg_map = self.sess.run(
    self.OUTPUT_TENSOR_NAME,
    feed_dict={self.INPUT_TENSOR_NAME: [np.asarray(resized_image)]})

newtime = time.time()
print(newtime-oldtime)

seg_map = batch_seg_map[0]
return resized_image, seg_map

`