Model trained with gpu is not as good as trained by cpu.

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [Yes ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not been filed already.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/official/...

2. Describe the bug

I want train a new model with ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8.

Model trained by CPU is better than GPU.

For the same picture, model trained by cpu can detect the object, but model from gpu can not.

gpu:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.186

cpu:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.286

3. Steps to reproduce

Steps to reproduce the behavior.

4. Expected behavior

Model trained by GPU should be the same as trained by CPU.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu20.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 2.8
• Python version: 3.8
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 8.1
• GPU model and memory: 16G