why multi-gpu training slower than single gpu

Describe the question.

I have rewritten my data augmentation methods using the DALI module and applied them to train the DeeplabV3 model based on TensorFlow. However, I have observed that the training speed is faster when using a single GPU, and the speed significantly decreases when training on 4 GPUs. Both my data augmentation methods and the generation of DaliDataset are implemented following the official documentation：https://docs.nvidia.com/deeplearning/dali/user-guide/docs/examples/frameworks/tensorflow/tensorflow-dataset-multigpu.html

My current concerns are as follows:

1. Why is the training slower with multiple GPUs, and even with a batch size set to 64, the GPU memory can still be fully utilized?
2. Does using the DALI data augmentation method really provide a noticeable speed improvement compared to using TensorFlow's native data augmentation method? Is it meaningful to validate the DALI approach?

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report