IoU for many classes in the evaluation are 0

System information

• What is the top-level directory of the model you are using: /mrtstorage/users/
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes, added some tf.Print in the files related to eval.py
  -OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18
• TensorFlow installed from (source or binary): N/A
• TensorFlow version (use command below): 1.14
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: N/A
• GPU model and memory: GeForce GTX TITAN X, used about 4GB of it
• Exact command to reproduce:

CUDA_VISIBLE_DEVICES="0" python deeplab/eval.py \
    --logtostderr \
    --eval_split="val" \
    --model_variant="xception_65" \
    --atrous_rates=6 \
    --atrous_rates=12 \
    --atrous_rates=18 \
    --output_stride=16 \
    --decoder_output_stride=4 \
    --eval_crop_size="1025,2049" \
    --dataset="cityscapes" \
    --checkpoint_dir=/mrtstorage/users/zwang/trainlog/19class_retrain_Monday \
    --eval_logdir=/mrtstorage/users/zwang/eval/19class_retrain_Tuesdaycheck \
    --dataset_dir=/mrtstorage/users/zwang/tfrecords/initial_tfrecord/

Ps. I am training the model from a checkpoint from model-zoo, with the initial encoder and decoder. So the network is supposed to work well even if I don't train it.

Describe the problem

Hi,

I was wondering why the miou of the eval.py is always the same (0.01982), no matter how long did I train the model. So I outputted some parameters from the evaluation process:

print the value of total_cm[[344433011 49242474 187481030 6324574 7220193 12835359 1429326 5898380 91260988 6829731 27329271 11450474 1843176 56439980 2560599 3375678 1006366 601945 6108830][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][56035 33991 6302418 65595 79678 389200 183486 144543 13217854 40545 1591879 90558 18248 347585 59480 29206 12109 4988 55653][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][704918 274511 6634315 324033 217836 319845 188015 65775 53624253 742176 199 284832 97587 2777095 95974 147377 11794 114232 304595][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][29 0 258392 0 3 1632 2 11 10225 0 1785080 0 0 47 34159 0 291 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][28017 8069 219118 6476 9316 18695 12920 1745 569573 13484 1645 64365 11532 194612 10257 11960 1540 7757 31774][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]

print the value of cm_diag[344433024 0 6302418 0 0 0 0 0 53624252 0 1.78508e+06 0 0 194612 0 0 0 0 0]

print the value of sum_over_row[3.45222e+08 49559044 200895280 6720678 7527026 13564731 1813749 6110454 158682896 7625936 30708074 11890229 1970543 59759320 2760469 3564221 1.0321e+06 728922 6500852]
print the value of sum_over_col[823671360 0 22723052 0 0 0 0 0 66929360 0 2089871 0 0 1222855 0 0 0 0 0]

print the value of denominator[824460416 49559044 217315920 6720678 7527026 13564731 1813749 6110454 1.71988e+08 7625936 31012864 11890229 1970543 60787564 2760469 3564221 1.0321e+06 728922 6500852]

print the value of all_iou[0.417767823 0 0.0290011801 0 0 0 0 0 0.311790645 0 0.0575593412 0 0 0.00320151 0 0 0 0 0]

the "total_cm" above represents the confusion matrix, of which the calculation code is not modified by me.
the "all_iou" above represents the IoU for each class in the dataset cityscapes (there are 19 classes)

the CS stands for the dataset of cityscapes in the tabel (from the masterarbeit of Mr.Frank Bieder).
Only the most popular classes are given a value in the confusion matrix (total_cm in the output abouve), and the not so regular classes are only just ignored in the evaluation process.
labal distributions of different class

The IoUs for each class are too small anyway, let alone there are so many zeros. But the it should work well even without training, and I can't figure out what could be wrong in the evaluation process. Could anyone help?