DeeplabV3+ is not converging with custom dataset

Prerequisites

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/deeplab

2. Describe the bug

I'm training deeplabv3+ on my custom two-class dataset using pre-trained network xception65_coco_voc_trainaug. Instead of the convergence model is deconverting, i.e. at the end(after a few iterations), the resulting image pixels are acquiring a value of 1 only. There is no class imbalance for sure. Even if it is, I've tried with multiple --label-weight values. But nothing got improved.

3. Steps to reproduce

I made a change in train_utils file present in utils folder at line 210.
Final:- exclude_list = ['global_step', 'logits'] (added logits)

In data_generator.py made changes for my custom 2 class data:-

I do have white edges to ignore, so total class=background+foreground=2
Mydata= DatasetDescriptor(
splits_to_sizes={
'train': 500,
'trainval':550,
'val': 50,
},
num_classes=2,
ignore_label=255,
)

_DATASETS_INFORMATION = {
'cityscapes': _CITYSCAPES_INFORMATION,
'pascal_voc_seg': _PASCAL_VOC_SEG_INFORMATION,
'ade20k': _ADE20K_INFORMATION,
'mydata':Mydata,
}

My input data is in RGB and target in gray-scale with the value of 0 for background and 1 for foregorund. Data-type is unit8 in both input and label.

The command I used for training is:

python train.py
--logtostderr
--training_number_of_steps=3000
--train_split="train"
--model_variant="xception_65"
--atrous_rates=12
--atrous_rates=24
--atrous_rates=36
--output_stride=8
--decoder_output_stride=4
--train_crop_size=512
--train_crop_size=512
--train_batch_size=1
--dataset="mydata"
--fine_tune_batch_norm=False
-- initialize_last_layer=False
--last_layers_contain_logits_only=True
--tf_initial_checkpoint='${some path}/model.ckpt'
--train_logdir='${some path}/train'
--dataset_dir='${some path}/tfrecord'

4. Expected behavior

A clear and concise description of what you expected to happen.

5. Additional context

Include any logs that would be helpful to diagnose the problem.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 16.04
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): conda 1.14-gpu
• Python version: 3.6
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:10
• GPU model and memory: