Performance comparability not possible?

System information

• What is the top-level directory of the model you are using: object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): model augmentation is removed from the networks .config file
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from (source or binary): binary, installed from pip: pip install tensorflow-gpu
• TensorFlow version (use command below): 1.14
• Bazel version (if compiling from source): None
• CUDA/cuDNN version: CUDA10
• GPU model and memory: GTX 1080
• Exact command to reproduce:

Describe the problem

I am writing my thesis about object detection with active learning using the tensorflow object detection api. Therefore i am trying to have a baseline setup from which i am trying to evaluate the effect of different parameter changes.

The problem: if i run the same experiment setup two times, the performance graph show (slightly) different results. Before each run i am reseting all values, a python seed is set, augmentation has been removed from the networks .config file.

It seems like there are still some sort of randomization happening within the network without having proper control /documentation about it. My guesses are:
-- there might be a random shuffle of the training images before each training?
-- dropout has some sort of random value embedded?

Reproducible results are very important for scientific work and right now really hard or not possible to achieve.

Source code / logs

I posted a question on stackoverflow already which shows the different graphs of the same experiment setup:
https://stackoverflow.com/questions/59032574/how-to-ensure-neural-net-performance-comparability