GPU acceleration only after first execution

When I'm trying to run model inferencing on GPU, the first execution time is much high than the others. Here, is my sample code:

import tensorflow as tf
import cv2
import numpy as np
import time

detection_graph1 = tf.Graph()
img = cv2.imread("/home/sumit/ved/3.jpg")
img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

with tf.device("/device:XLA_GPU:0"):
    with detection_graph1.as_default():
        tf_config = tf.ConfigProto()
        tf_config.gpu_options.allow_growth = True
        tf_sess1 = tf.Session(config=tf_config, graph=detection_graph1)
        model = tf.saved_model.loader.load(tf_sess1, ["serve"], "/home/sumit/ved/saved_model")

    tf_input = tf_sess1.graph.get_tensor_by_name('image_tensor:0')
    tf_scores = tf_sess1.graph.get_tensor_by_name('detection_scores:0')
    tf_boxes = tf_sess1.graph.get_tensor_by_name('detection_boxes:0')
    tf_classes = tf_sess1.graph.get_tensor_by_name('detection_classes:0')
    tf_num_detections = tf_sess1.graph.get_tensor_by_name('num_detections:0')

    for i in range(5):
        tic = time.time()
        scores, boxes, classes, num_detections = tf_sess1.run([tf_scores, tf_boxes, tf_classes, tf_num_detections], 
                                                                  feed_dict={tf_input: img[None, ...]})
        print("Time taken: ", time.time()-tic)

Here, is the output:

Time taken:  4.89597749710083
Time taken:  0.1528186798095703
Time taken:  0.13585400581359863
Time taken:  0.13776516914367676
Time taken:  0.13747143745422363

If I run the code again, I get the same output. Speed of first execution is very much important for me, since I've to do inferencing on individual images sent through the network. After first execution the system might me idle for sometimes.

Here, is my system configurations:
CUDA: 10.0
Tensorflow version: 1.14.0
GPU: NVIDIA Tesla M60