Number of AvgNumGroundtruthBoxesPerImage is always 100

System information

• What is the top-level directory of the model you are using: object_detection/models/research/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Kubuntu 18.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.14
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: Cuda 10.0 cuDnn7
• GPU model and memory: RTX 2080TI
• Exact command to reproduce:

PIPELINE_CONFIG_PATH=/home/lanius/code/object_detection/trained_models/ssd_resnet50_v1_fpn_shared_box_predictor/pipeline_clean_few_classes.config
MODEL_DIR=/home/lanius/code/object_detection/results/ssd_resnet50_v1_fpn/clean_few_classes
SAMPLE_1_OF_N_EVAL_EXAMPLES=1
python object_detection/model_main.py \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --model_dir=${MODEL_DIR} \
    --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
    --alsologtostderr 

Describe the problem

I am attempting to train the ssd_resnet_50_fpn_coco object detector on a simplified coco dataset, however the same issue persists even if I use the all coco classes. The metrics reported by tensorboard look off:

The number of AvgNumGroundtruthBoxesPerImage is always 100, the maximum number of output boxes from the model.
Indeed in the tensorboard display, there are some images with no GT bboxes:

While the loss in general decreases, the performance of the detector is very poor. After 15000 steps with a batchsize of 8:

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.015
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.031
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.015
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.031
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.018
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.025
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.026
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.006
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.059