multi GPU training takes longer than single GPU training

I am trying to train yolox_tiny on a custom dataset. I have 8 NVIDIA GPUs from AWS's p3.16xlarge instance. I tried to do small number of epochs training to see how it goes. Looks like the multi GPU training takes longer than single GPU training - 4 hr vs 2 hr for max_epoch = 10

command for 8 GPU training

python tools/train.py -f exps/example/custom/yolox_tiny.py -d 8 -b 64 --fp16 -o -c yolox_tiny.pth --logger wandb wandb-project yolox_tiny wandb-id 1

Command for 1 GPU training

python tools/train.py -f exps/example/custom/yolox_tiny.py -d 1 -b 8 --fp16 -o -c yolox_tiny.pth --logger wandb wandb-project yolox_tiny wandb-id 2

I tested it both with yolox 0.3.0 and 0.2.0

Here is the exp file

import os

from yolox.exp import Exp as MyExp


class Exp(MyExp):
    def __init__(self):
        super(Exp, self).__init__()
        self.depth = 0.33
        self.width = 0.375
        self.input_size = (416, 416)
        self.random_size = (10, 20)
        self.test_size = (416, 416)
        self.exp_name = os.path.split(os.path.realpath(__file__))[1].split(".")[0]
        
        # --------------- transform config ----------------- #
        self.mosaic_prob = 1.0
        self.mixup_prob = 1.0
        self.hsv_prob = 1.0
        self.flip_prob = 0.5
        self.degrees = 10.0
        self.translate = 0.1
        self.scale = (0.1, 2)  
        self.mosaic_scale = (0.5, 1.5)
        self.mixup_scale = (0.5, 1.5)
        self.shear = 2.0
        self.perspective = 0.0
        self.enable_mixup = True   # default is False, if true longer time to train
        
        # -------- Training config -----#
        self.warmup_epochs = 1   
        self.no_aug_epochs = 5   
        
        # Define yourself dataset path
        self.data_dir = "datasets/my_dataset"
        self.train_ann = "instances_train2017.json"
        self.val_ann = "instances_val2017.json"

        self.num_classes = 2

        self.max_epoch = 10
        self.data_num_workers = 4
        self.eval_interval = 1

Start with tools/train.py and exps/example/custom/yolox_tiny.py, then reproduce the two commands using the supplied batch sizes and eight-versus-one GPU settings. Compare per-epoch timing, data-loading behavior, evaluation intervals, and augmentation settings. Done means identifying the source of the multi-GPU slowdown and documenting or correcting the relevant behavior with evidence.