[object_detection] non-determinism within the training

System information

• What is the top-level directory of the model you are using: models-master/research/object_detection/
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes but mostly print statements inside the core code
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from (source or binary): binary, installed from pip: pip install tensorflow-gpu
• TensorFlow version (use command below): 1.14
• Bazel version (if compiling from source):
• CUDA/cuDNN version: CUDA Version: 10.0
• GPU model and memory: GeForce GTX 1080
• Exact command to reproduce: python train.py --logtostderr --train_dir=...

Describe the problem

Running training twice with the same checkpoint as starting point results in different weights after the first step already. (The weights are the same after initialization from checkpoint 0).

Example:
Experiment A:

• weights at checkpoint 0: 2230.3213
• weights after the first step of traninig: 2230.3845

Experiment B:

• weights at checkpoint 0: 2230.3213
• weights after the first step of traninig: 2230.3768

Therefore i am suspecting that the training process (gpu) is having some non-determinism?

This results in incomparable results between many runs and parameter changes and makes scientific work barely possible.