pascal_voc_detection_metrics gives very low scores for first category in label_map

I am following this tutorial to train a Faster R-CNN model for object detection on my own data.

When evaluating using python model_main_tf2.py --model_dir=models/MY_MODEL --pipeline_config_path=models/MY_MODEL/pipeline.config --checkpoint_dir=models/MY_MODEL and Pascal VOC evaluator the first category in my label map is never evaluated correctly.

I have trained the modell for a different number of classes, every time the first category has very low mAP.
The code I'm posting here refers to a model trained for object detection for two classes. The record files have been created with the script from the tutorial.

This is my config file:

model {
  faster_rcnn {
    num_classes: 2
    image_resizer {
      fixed_shape_resizer {
        width: 1024
        height: 1024
      }
    }
    feature_extractor {
      type: 'faster_rcnn_resnet101_keras'
      batch_norm_trainable: true
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 16
        width_stride: 16
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        share_box_across_classes: true
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 300
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
    use_static_shapes: true
    use_matmul_crop_and_resize: true
    clip_anchors_to_image: true
    use_static_balanced_label_sampler: true
    use_matmul_gather_in_matcher: true
  }
}
train_config {
  batch_size: 4
  num_steps: 6000
  sync_replicas: true
  optimizer {
    momentum_optimizer {
      learning_rate {
        cosine_decay_learning_rate {
          learning_rate_base: 0.002
          total_steps: 6000
          warmup_learning_rate: 0.0002
          warmup_steps: 300
          hold_base_rate_steps: 0
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  fine_tune_checkpoint: "pre-trained-models/faster_rcnn_resnet101_v1_1024x1024_coco17_tpu-8/checkpoint/ckpt-0"
  startup_delay_steps: 0.0
  replicas_to_aggregate: 8
  max_number_of_boxes: 100
  unpad_groundtruth_tensors: false
  fine_tune_checkpoint_type: "detection"
  use_bfloat16: false
  fine_tune_checkpoint_version: V2
}
train_input_reader: {
  label_map_path: "annotations/label_map.pbtxt"
  tf_record_input_reader {
    input_path: "annotations/train.record"
  }
}
eval_config: {
  metrics_set: "pascal_voc_detection_metrics"
  use_moving_averages: false
}
eval_input_reader: {
  label_map_path: "annotations/label_map.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "annotations/test.record"
  }
}

My label_map:
`item {
    id: 1
    name: 'Signal'
}
item {
    id: 2
    name: 'Schild' # this means sign in german
}

The results from evaluation are

I0419 13:21:50.826318 140148733331264 object_detection_evaluation.py:1335] average_precision: 0.004911
I0419 13:21:50.853986 140148733331264 object_detection_evaluation.py:1335] average_precision: 0.802144
INFO:tensorflow:Eval metrics at step 6000
I0419 13:21:50.858681 140148733331264 model_lib_v2.py:988] Eval metrics at step 6000
INFO:tensorflow:        + PascalBoxes_Precision/mAP@0.5IOU: 0.403527
I0419 13:21:50.907640 140148733331264 model_lib_v2.py:991]      + PascalBoxes_Precision/mAP@0.5IOU: 0.403527
INFO:tensorflow:        + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Signal: 0.004911
I0419 13:21:50.909214 140148733331264 model_lib_v2.py:991]      + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Signal: 0.004911
INFO:tensorflow:        + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Schild: 0.802144
I0419 13:21:50.910397 140148733331264 model_lib_v2.py:991]      + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Schild: 0.802144
INFO:tensorflow:        + Loss/RPNLoss/localization_loss: 0.009531
I0419 13:21:50.911445 140148733331264 model_lib_v2.py:991]      + Loss/RPNLoss/localization_loss: 0.009531
INFO:tensorflow:        + Loss/RPNLoss/objectness_loss: 0.003197
I0419 13:21:50.912504 140148733331264 model_lib_v2.py:991]      + Loss/RPNLoss/objectness_loss: 0.003197
INFO:tensorflow:        + Loss/BoxClassifierLoss/localization_loss: 0.022416
I0419 13:21:50.913525 140148733331264 model_lib_v2.py:991]      + Loss/BoxClassifierLoss/localization_loss: 0.022416
INFO:tensorflow:        + Loss/BoxClassifierLoss/classification_loss: 0.027070
I0419 13:21:50.914626 140148733331264 model_lib_v2.py:991]      + Loss/BoxClassifierLoss/classification_loss: 0.027070
INFO:tensorflow:        + Loss/regularization_loss: 0.000000
I0419 13:21:50.915687 140148733331264 model_lib_v2.py:991]      + Loss/regularization_loss: 0.000000
INFO:tensorflow:        + Loss/total_loss: 0.062212
I0419 13:21:50.916763 140148733331264 model_lib_v2.py:991]      + Loss/total_loss: 0.062212

Experiments:

• When using these categories seen below, it is Masts that have very low mAP@0.5 at 0.025

item {
    id: 1
    name: 'Mast'
}

item {
    id: 2
    name: 'Signal'
}

item {
    id: 3
    name: 'Schild'
}

• Using a dummy category (which has no objects in the data) as first category in the label map, all other categories are evaluated in a meaningful way (the overall mAP is of course affected by the low mAP of the dummy class)

item {
    id: 1
    name: 'Dummy'
}

item {
    id: 2
    name: 'Mast'
}

item {
    id: 3
    name: 'Signal'
}

item {
    id: 4
    name: 'Schild'
}

Evaluation results:

I0420 10:52:29.358691 140207870719808 checkpoint_utils.py:134] Found new checkpoint at models/my_faster_rcnn_resnet101_1024_test1/ckpt-5
INFO:tensorflow:Finished eval step 0
I0420 10:52:42.201465 140207870719808 model_lib_v2.py:939] Finished eval step 0
I0420 10:53:19.393145 140207870719808 object_detection_evaluation.py:1335] average_precision: 0.000000
I0420 10:53:19.398215 140207870719808 object_detection_evaluation.py:1335] average_precision: 0.198166
I0420 10:53:19.401919 140207870719808 object_detection_evaluation.py:1335] average_precision: 0.621884
I0420 10:53:19.405840 140207870719808 object_detection_evaluation.py:1335] average_precision: 0.692288
INFO:tensorflow:Eval metrics at step 1500
I0420 10:53:19.407017 140207870719808 model_lib_v2.py:988] Eval metrics at step 1500
INFO:tensorflow:        + PascalBoxes_Precision/mAP@0.5IOU: 0.378084
I0420 10:53:19.435913 140207870719808 model_lib_v2.py:991]      + PascalBoxes_Precision/mAP@0.5IOU: 0.378084
INFO:tensorflow:        + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Dummy: 0.000000
I0420 10:53:19.436885 140207870719808 model_lib_v2.py:991]      + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Dummy: 0.000000
INFO:tensorflow:        + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Mast: 0.198166
I0420 10:53:19.437575 140207870719808 model_lib_v2.py:991]      + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Mast: 0.198166
INFO:tensorflow:        + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Signal: 0.621884
I0420 10:53:19.438186 140207870719808 model_lib_v2.py:991]      + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Signal: 0.621884
INFO:tensorflow:        + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Schild: 0.692288
I0420 10:53:19.438765 140207870719808 model_lib_v2.py:991]      + PascalBoxes_PerformanceByCategory/AP@0.5IOU/Schild: 0.692288
INFO:tensorflow:        + Loss/RPNLoss/localization_loss: 0.038080
I0420 10:53:19.439270 140207870719808 model_lib_v2.py:991]      + Loss/RPNLoss/localization_loss: 0.038080
INFO:tensorflow:        + Loss/RPNLoss/objectness_loss: 0.013058
I0420 10:53:19.439779 140207870719808 model_lib_v2.py:991]      + Loss/RPNLoss/objectness_loss: 0.013058
INFO:tensorflow:        + Loss/BoxClassifierLoss/localization_loss: 0.036514
I0420 10:53:19.440281 140207870719808 model_lib_v2.py:991]      + Loss/BoxClassifierLoss/localization_loss: 0.036514
INFO:tensorflow:        + Loss/BoxClassifierLoss/classification_loss: 0.046075
I0420 10:53:19.440786 140207870719808 model_lib_v2.py:991]      + Loss/BoxClassifierLoss/classification_loss: 0.046075
INFO:tensorflow:        + Loss/regularization_loss: 0.000000
I0420 10:53:19.441344 140207870719808 model_lib_v2.py:991]      + Loss/regularization_loss: 0.000000
INFO:tensorflow:        + Loss/total_loss: 0.133727
I0420 10:53:19.441860 140207870719808 model_lib_v2.py:991]      + Loss/total_loss: 0.133727

• ALL categories can be found using inference on test images
• Evaluating with COCO metrics it gives me an mAP@0.50IOU of 0.773897

By now I think this might be a bug, but I'm hoping it is a problem in my code/config/data that I can solve.

Can somebody help me to solve this issue?
Does it have to do with category ID 1 interfering with background category (ID 0)?