mIoU value for some classes for Xception65 model remains at 0 during training

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [n] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [ y] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [ y] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/deeplab/train.py
https://github.com/tensorflow/models/tree/master/research/deeplab/eval.py

2. Describe the bug

I train Xception65 model pretrained on ImageNet (i.e. xception_65_imagenet from DeepLab Model Zoo) on Cityscapes Trainfine dataset. When I ran eval.py on the most recent checkpoint saved during the training, mIoU for some of the classes remains at 0, namely:
eval/miou_1.0_class_6[0]
eval/miou_1.0_class_12[0]
eval/miou_1.0_class_16[0]
eval/miou_1.0_class_11[0.510199]
eval/miou_1.0_class_17[0]
eval/miou_1.0_class_4[0.111635856]
eval/miou_1.0_class_8[0.814882576]
eval/miou_1.0_class_0[0.891150713]
eval/miou_1.0_class_14[0]
eval/miou_1.0_class_15[0]
eval/miou_1.0_class_18[0.383434713]
eval/miou_1.0_overall[0.350808531]
eval/miou_1.0_class_10[0.835741937]
eval/miou_1.0_class_2[0.773912489]
eval/miou_1.0_class_5[0.189518735]
eval/miou_1.0_class_13[0.827344835]
eval/miou_1.0_class_7[0.259665072]
eval/miou_1.0_class_1[0.665363967]
eval/miou_1.0_class_9[0.402512431]
eval/miou_1.0_class_3[0]

3. Steps to reproduce

I run train.py script with the following parameters:
python deeplab/train.py
--logtostderr
--training_number_of_steps=150000
--train_split="train_fine"
--model_variant="xception_65"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=16
--decoder_output_stride=4
--train_crop_size="769,769"
--train_batch_size=2
--dataset="cityscapes"
--tf_initial_checkpoint=${PATH_TO_INITIAL_CHECKPOINT}
--train_logdir=${PATH_TO_TRAIN_DIR}
--fine_tune_batch_norm=True
--dataset_dir=${PATH_TO_DATASET}
--save_interval_secs=300 \
--initialize_last_layer=False
--last_layers_contain_logits_only=True

and subsequently eval.py with the following parameters:
python deeplab/eval.py
--logtostderr
--eval_split="val"
--model_variant="xception_65"
--atrous_rates=6
--atrous_rates=12
--atrous_rates=18
--output_stride=16
--decoder_output_stride=4
--eval_crop_size="1025,2049"
--dataset="cityscapes"
--checkpoint_dir=${PATH_TO_CHECKPOINT}
--eval_logdir=${PATH_TO_EVAL_DIR}
--dataset_dir=${PATH_TO_DATASET}
--max_number_of_evaluations=1

4. Expected behavior

I would expect that after 150 000 iterations, all of the classes would have mIoU > 0.

5. Additional context

None.

6. System information

• OS Platform and Distribution: Linux Ubuntu 16.04
• TensorFlow installed from binary
• TensorFlow version 1.15.0 (use command below):
• Python version 3.5
• CUDA/cuDNN version: 10.2
• GPU model and memory: NVIDIA Tesla K40c, 11441MiB