[Deeplab] mIOU increasing but training loss curve all over the place.

System information

• What is the top-level directory of the model you are using: models/research/deeplab
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.2 LTS
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): 1.13.1
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 10.1
• GPU model and memory:
• Exact command to reproduce:

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

I trained the xception_65 model for 150000 iterations on a custom dataset. As the training kept going, validation improved.
Iterations 50k -> (mIOU: 0.95)
Iterations 100k -> (mIOU: 0.96)
........mIOU: (0.965) for 150k

But the training loss is all over the place. I don't understand why is this happening. Any leads?

Source code / logs

NUM_ITERATIONS=150000
python3 "${WORK_DIR}"/train.py
--logtostderr
--train_split="trainval"
--model_variant="xception_65"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=8
--decoder_output_stride=4
--train_crop_size="321,321"
--dataset="lake"
--train_batch_size=8
--training_number_of_steps="${NUM_ITERATIONS}"
--fine_tune_batch_norm=false
--train_logdir="${TRAIN_LOGDIR}"
--base_learning_rate=0.0001
--learning_policy="poly"
--tf_initial_checkpoint="training from pascal VOC pretrained checkpoint"
--dataset_dir="${DATASET}"