[DeepLab] Add Early Stopping support

System information

• What is the top-level directory of the model you are using: tensorflow/models/research/deeplab
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): b'unknown' 1.13.1
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 10.1
• GPU model and memory: 1080ti
• Exact command to reproduce:

python "${WORK_DIR}"/train.py \
  --logtostderr \
  --train_split="train" \
  --model_variant="xception_65" \
  --num_clones=2 \
  --atrous_rates=6 \
  --atrous_rates=12 \
  --atrous_rates=18 \
  --output_stride=16 \
  --decoder_output_stride=4 \
  --train_crop_size="513,513" \
  --train_batch_size=4 \
  --training_number_of_steps="${NUM_ITERATIONS}" \
  --fine_tune_batch_norm=true \
  --tf_initial_checkpoint="${INIT_FOLDER}/model.ckpt" \
  --train_logdir="${TRAIN_LOGDIR}" \
  --dataset_dir="${DATASET_DIR}"

Describe the problem

Loss is increasing drastically after a specific number of steps when transfer learning. To make training easier I propose early stopping is implemented and added as an option.

Source code / logs