object_detection tensorflow 1 models not correctly fine-tuned during training

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md

2. Describe the bug

I am trying to train a model by fine-tuning from a pre-trained tf1 model from tf1_detection_zoo. However, when I start the training process and initiate evaluation with the newly generated training checkpoints, the model makes zero detections. I printed out the weights of the model and it appears that they are default values and not fine-tuned from the pre-trained model. Specifically, I am fine-tuning from the faster_rcnn_resnet50_coco model.

3. Steps to reproduce

I am using tensorflow version 1.15

For the pipeline.config files I used the sample configs from the tensorflow repository (where I changed the paths to match my own directory):

model {
faster_rcnn {
num_classes: 90
image_resizer {
keep_aspect_ratio_resizer {
min_dimension: 600
max_dimension: 1024
}
}
feature_extractor {
type: "faster_rcnn_resnet50"
first_stage_features_stride: 16
}
first_stage_anchor_generator {
grid_anchor_generator {
height_stride: 16
width_stride: 16
scales: 0.25
scales: 0.5
scales: 1.0
scales: 2.0
aspect_ratios: 0.5
aspect_ratios: 1.0
aspect_ratios: 2.0
}
}
first_stage_box_predictor_conv_hyperparams {
op: CONV
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
truncated_normal_initializer {
stddev: 0.00999999977648
}
}
}
first_stage_nms_score_threshold: 0.0
first_stage_nms_iou_threshold: 0.699999988079
first_stage_max_proposals: 100
first_stage_localization_loss_weight: 2.0
first_stage_objectness_loss_weight: 1.0
initial_crop_size: 14
maxpool_kernel_size: 2
maxpool_stride: 2
second_stage_box_predictor {
mask_rcnn_box_predictor {
fc_hyperparams {
op: FC
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
variance_scaling_initializer {
factor: 1.0
uniform: true
mode: FAN_AVG
}
}
}
use_dropout: false
dropout_keep_probability: 1.0
}
}
second_stage_post_processing {
batch_non_max_suppression {
score_threshold: 0.300000011921
iou_threshold: 0.600000023842
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SOFTMAX
}
second_stage_localization_loss_weight: 2.0
second_stage_classification_loss_weight: 1.0
}
}
train_config {
batch_size: 1
data_augmentation_options {
random_horizontal_flip {
}
}
optimizer {
momentum_optimizer {
learning_rate {
manual_step_learning_rate {
initial_learning_rate: 0.000300000014249
schedule {
step: 0
learning_rate: 0.000300000014249
}
schedule {
step: 900000
learning_rate: 2.99999992421e-05
}
schedule {
step: 1200000
learning_rate: 3.00000010611e-06
}
}
}
momentum_optimizer_value: 0.899999976158
}
use_moving_average: false
}
gradient_clipping_by_norm: 10.0
fine_tune_checkpoint: "PATH_TO_BE_CONFIGURED/model.ckpt"
from_detection_checkpoint: true
num_steps: 200000
}
train_input_reader {
label_map_path: "PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt"
tf_record_input_reader {
input_path: "PATH_TO_BE_CONFIGURED/mscoco_train.record"
}
}
eval_config {
num_examples: 8000
max_evals: 10
use_moving_averages: false
}
eval_input_reader {
label_map_path: "PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt"
shuffle: false
num_readers: 1
tf_record_input_reader {
input_path: "PATH_TO_BE_CONFIGURED/mscoco_val.record"
}
}

I execute this python script:
python model_main.py
--model_dir={path to model directory}
--pipeline_config_path={path to config}/pipeline.config
--alsologtostderr

When the new checkpoints have been created in the model directory and after the first training step has finished, I stop the training and start the evaluation process.

To print the weights of the model checkpoints, I insert the following code before the evaluation process has started (before "if FLAGS.checkpoint_dir:"). I also printed the weights for the pre-trained model too.

estimator_var_names = estimator.get_variable_names()
print("estimator variable values:")
for var_name in estimator_var_names:
	print("variable name: " + var_name)
	print(estimator.get_variable_value(var_name))

I execute the script again for evaluation:
python model_main.py
--model_dir={path to model directory}
--pipeline_config_path={path to config}/pipeline.config
--checkpoint_dir={path to model directory that contains newly created model.ckpt files}

During evaluation, the model makes zero detections. Also, the weights of the newly generated checkpoint are different from the weights of the pre-trained model.

4. Expected behavior

I expect the accuracy and weights of the newly generated checkpoints to be roughly similar to the accuracy and weights of the pre-trained model.

5. Additional context

In addition to the pipeline.config I provided above, I also tried:
fine_tune_checkpoint_type: "detection"
instead of
from_detection_checkpoint: true
but the same problem occurs.

This bug happens on both Google Colab and my local machine.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab
• Mobile device name if the issue happens on a mobile device: N/A
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.15
• Python version: 3.7.10
• Bazel version (if compiling from source): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: Using Google Colab
• GPU model and memory: Using Google Colab