Deeplab - num_clones unexpected convergence behaviour

System information

• What is the top-level directory of the model you are using:
  tensorflow-models/research/deeplab

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  I did

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Ubuntu 16.04.6 LTS

• TensorFlow installed from (source or binary):
  Installed from binary

• TensorFlow version (use command below):
  Tensorflow 1.13.1

• Bazel version (if compiling from source):

• CUDA/cuDNN version:
  7.4.2

• GPU model and memory:
  NVidia Tesla P100, 16280MiB

• Exact command to reproduce:
  //

Describe the problem

There are some differences between convergence on runs done with num_clones>1 and runs done on a single GPU even if the minibatch size is the same. I have isolated this problem by training the same model on the same dataset with exactly the same configurations with minibatch 6 on a Tesla P100 with no clones and with 3 clones. These are the results of, for example, the regularization loss:

The first image is the regularization loss with 3 clones, the second one is without clones. Their convergence pattern doesn't match.
Taking another experiment in comparison: in the following two runs, the configurations at training time are exactly the same (only 1 clone) and the only thing that changes between the two is that the dataset is preprocessed by a simple script I made:

As evident, they are very close to each other even if the data was preprocessed in one and the other not.
Did anyone else experience this problem?

@aquariusjay

This is the script I use to train and monitor without clones (with clones I just add --num_clones=3):

python deeplab/train.py \
    --logtostderr \
    --training_number_of_steps=1000 \
    --train_split="train" \
    --model_variant="xception_65" \
    --base_learning_rate=0.0001 \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --train_crop_size='513,513' \
    --train_batch_size=6 \
    --dataset=<my_data_name> \
    --tf_initial_checkpoint="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/pretrained/deeplabv3_pascal_trainval/model.ckpt" \
    --train_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --dataset_dir=<data_dir>

python deeplab/eval.py \
    --logtostderr \
    --eval_split="val" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --eval_crop_size='1201,1201' \
    --dataset=<my_data>\
    --checkpoint_dir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --eval_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs/eval" \
    --dataset_dir=<my_data>

python deeplab/train.py \
    --logtostderr \
    --training_number_of_steps=2000 \
    --train_split="train" \
    --model_variant="xception_65" \
    --base_learning_rate=0.0001 \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --train_crop_size='513,513' \
    --train_batch_size=6 \
    --dataset=<my_data_name> \
    --tf_initial_checkpoint="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/pretrained/deeplabv3_pascal_trainval/model.ckpt" \
    --train_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --dataset_dir=<data_dir>

python deeplab/eval.py \
    --logtostderr \
    --eval_split="val" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --eval_crop_size='1201,1201' \
    --dataset=<my_data>\
    --checkpoint_dir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --eval_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs/eval" \
    --dataset_dir=<my_data>

python deeplab/train.py \
    --logtostderr \
    --training_number_of_steps=3000 \
    --train_split="train" \
    --model_variant="xception_65" \
    --base_learning_rate=0.0001 \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --train_crop_size='513,513' \
    --train_batch_size=6 \
    --dataset=<my_data_name> \
    --tf_initial_checkpoint="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/pretrained/deeplabv3_pascal_trainval/model.ckpt" \
    --train_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --dataset_dir=<data_dir>

python deeplab/eval.py \
    --logtostderr \
    --eval_split="val" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --eval_crop_size='1201,1201' \
    --dataset=<my_data>\
    --checkpoint_dir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --eval_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs/eval" \
    --dataset_dir=<my_data>

python deeplab/train.py \
    --logtostderr \
    --training_number_of_steps=4000 \
    --train_split="train" \
    --model_variant="xception_65" \
    --base_learning_rate=0.0001 \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --train_crop_size='513,513' \
    --train_batch_size=6 \
    --dataset=<my_data_name> \
    --tf_initial_checkpoint="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/pretrained/deeplabv3_pascal_trainval/model.ckpt" \
    --train_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --dataset_dir=<data_dir>

python deeplab/eval.py \
    --logtostderr \
    --eval_split="val" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --eval_crop_size='1201,1201' \
    --dataset=<my_data>\
    --checkpoint_dir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --eval_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs/eval" \
    --dataset_dir=<my_data>

python deeplab/train.py \
    --logtostderr \
    --training_number_of_steps=5000 \
    --train_split="train" \
    --model_variant="xception_65" \
    --base_learning_rate=0.0001 \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --train_crop_size='513,513' \
    --train_batch_size=6 \
    --dataset=<my_data_name> \
    --tf_initial_checkpoint="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/pretrained/deeplabv3_pascal_trainval/model.ckpt" \
    --train_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --dataset_dir=<data_dir>

python deeplab/eval.py \
    --logtostderr \
    --eval_split="val" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --eval_crop_size='1201,1201' \
    --dataset=<my_data>\
    --checkpoint_dir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs" \
    --eval_logdir="$(echo ~)/semantic_segmentation/tensorflow-models/research/deeplab/trainlogs/eval" \
    --dataset_dir=<my_data>

With --num_clones=3 it is using 3 clones indeed and nvidia-smi looks like this:
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 46390 C python 8679MiB |
| 1 46390 C python 9191MiB |
| 2 46390 C python 9191MiB |
| 3 46390 C python 281MiB |
+-----------------------------------------------------------------------------+