Creating a new dataset from the Pascal dataset and adding one class fails to recognize the new class

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ x ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [x ] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [ x ] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/...

2. Describe the bug

Hey I am attempting to add a new class to the pascal dataset to make it 21 detectable classes. I have followed these steps:

1. Segment Images. I picked chickens and made 4 separate chicken segementations. Here's an example:
   enter image description here enter image description here enter image description here

2. place corresponding images in Voc2012 SegementationClass, SegmentationClassRaw, SegmentationObject, and JPEGImages

3. Add Annotation XML

4. Add new image to all txt files in "VOC2012\ImageSets\Main"

5. Add 3 new txt files in "VOC2012\ImageSets\Main": chicken_train.txt, chicken_trainval.txt, and chicken_val.txt. Add a 1 for the file name with chickens in it. Add 2 chicken files to VOC2012\ImageSets\Segmentation\train.txt and 2 others to trainval + val.

6. Create a new dataset in "data_generator.py" called "extended":

_EXTENDED_SEG_INFORMATION = DatasetDescriptor( splits_to_sizes={ 'train': 1466, 'train_aug': 10583, 'trainval': 2915, 'val': 1451, }, num_classes=22, ignore_label=255, )

7. create a new colormap for "extended" dataset:

_ADE20K = 'ade20k'
_CITYSCAPES = 'cityscapes'
_MAPILLARY_VISTAS = 'mapillary_vistas'
_PASCAL = 'pascal'
_EXTENDED = 'extended'

elif dataset == _EXTENDED:
return create_extended_label_colormap()

_DATASET_MAX_ENTRIES = {
_ADE20K: 151,
_CITYSCAPES: 256,
_MAPILLARY_VISTAS: 66,
_PASCAL: 512,
_EXTENDED: 512
}

def create_extended_label_colormap(): return np.asarray([ [0, 0, 0], #background [128, 0, 0], #aeroplane [0, 128, 0], #bike [128, 128, 0], #bird [0, 0, 128], #boat [128, 0, 128], #bottle [0, 128, 128], #bus [128, 128, 128], #car [64, 0, 0], #cat [192, 0, 0], #chair [64, 128, 0], #cow [64, 0, 128], #diningtable [64, 0, 128], #dog [192, 0, 128], #horse [64, 128, 128], #motorbike [192, 128, 128], #person [0, 64, 0], #pottedplant [128, 64, 0], #sheep [0, 192, 0], #sofa [128, 192, 0], #train [0, 64, 128], #tv [128, 64, 128], #chicken ])

8. Generate TFRecord with command:

py "research\deeplab\datasets\build_voc2012_data.py" --image_folder="D:\DeepLab\models-master\research\deeplab\datasets\pascal_voc_seg\VOCdevkit\VOC2012\JPEGImages" --semantic_segmentation_folder="research\deeplab\datasets\pascal_voc_seg\VOCdevkit\VOC2012\SegmentationClassRaw" --list_folder="research\deeplab\datasets\pascal_voc_seg\VOCdevkit\VOC2012\ImageSets\Segmentation" --image_format="jpg" --output_dir="research\deeplab\datasets\extended\tfrecord"

9. Artificially, set weights towards new chicken class to get it to show up. Train with command:

python "D:\DeepLab\models-master\research\deeplab\train.py" --logtostderr --train_split="trainval" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --train_crop_size="513,513" --train_batch_size=2 --dataset="extended" --training_number_of_steps=1000 --fine_tune_batch_norm=true --tf_initial_checkpoint="D:\DeepLab\models-master\research\deeplab\datasets\pascal_voc_seg\init_models\deeplabv3_pascal_train_aug\model.ckpt" --train_logdir="D:\DeepLab\models-master\research\deeplab\datasets\extended\exp\train_on_trainval_set\train" --dataset_dir="D:\DeepLab\models-master\research\deeplab\datasets\extended\tfrecord" --label_weights=.001 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=.01 --label_weights=100

10. exporting the model and testing it give me this which isn't great:

eval command on the image also produces poor results:

eval/miou_1.0_class_1[0.00715755112] eval/miou_1.0_class_16[0.00142123573]eval/miou_1.0_class_17[0.000168795494] eval/miou_1.0_class_12[0] eval/miou_1.0_class_14[0] eval/miou_1.0_class_0[0.126532018] eval/miou_1.0_class_5[0] eval/miou_1.0_class_13[0.0135749253] eval/miou_1.0_class_15[0.0179808438] eval/miou_1.0_class_3[0.000226198768] eval/miou_1.0_class_8[0.229160205]

eval/miou_1.0_class_2[8.26003941e-07] eval/miou_1.0_class_7[0.000108588596] eval/miou_1.0_class_4[0] eval/miou_1.0_class_19[0] eval/miou_1.0_class_18[0.00898866542] eval/miou_1.0_class_6[4.23762758e-05] eval/miou_1.0_overall[0.0191962905] eval/miou_1.0_class_21[0] eval/miou_1.0_class_10[0.00145707454] eval/miou_1.0_class_9[4.08720407e-05] eval/miou_1.0_class_20[0.00886027142] eval/miou_1.0_class_11[0.00659790076]

I have tried training 10,000 iterations to see if that fixes it and it does populate every other class but chicken class (class_21) still stays at 0 miou. What am I doing wrong here?!?

ADDITIONAL NOTES:
The biggest clue I have right now is if I modify or EVEN REMOVE the "get_dataset_colormap.py" file which supposedly determines which color corresponds to which dataset, nothing happens. I cannot hit prints in that file either so I am suspicious that the new chicken class color just isn't being picked up as relevant to my dataset.

I do have everything working correctly if I stick to the base pascal classes in my new "extended" dataset. I can get a pretty good model at like .81 miou overall. Issues only show up when I try to introduce the new class. I have tried training it with and without the label_weights, neither made a difference in finding the last chicken 21st class.

I am only adding 4 new chicken images so I don't expect the result to be good but i do expect it to exist :/. I am adding 2 into training and 2 into training/val.

I am unsure if i need to do another to substitute out "pascal_label_map" in "research\object_detection\data" or if that is unneeded.

3. Steps to reproduce

Add a new class to pascal in a new dataset

4. Expected behavior

class is recognizable

5. Additional context

none. see ADDITIONAL NOTES above.

6. System information

Windows 10
TensorFlow installed from Anaconda
TensorFlow version 2.20
Python version 3.7.6
CUDA/cuDNN version: 7.6.5
GPU model and memory: Nvidia GeForce RTX 2080 Super, total memory: 161766 MB, display memory 8010 MB , shared memory 8166 MB