learning_rate is constantly zero after restarting training from a previous run with EfficientDet D4

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [x ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [ x] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [ x] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection/configs/tf2/ssd_efficientdet_d4_1024x1024_coco17_tpu-32.config.

2. Describe the bug

I trained a EfficientDet D4 using "detection" for the fine_tune_checkpoint_type option to build a model for custom dataset. I wanted to continue the training so I changed three things:

1. Changed fine_tune_checkpoint_type option to "full' so that I can retain the detection head
2. Changed fine_tune_checkpoint to the latest checkpoint from the previous model build
3. Ran training again with slightly updated momentum_optimizer parameters.

When training with this new run, the learning_rate stays at 0 (confirmed from the console output and the tensorboard), and furthermore the regularization_loss is constant (as the model isn't updating any of it's weights)

3. Steps to reproduce

The relevant portion of the config file is below

train_config: {
  fine_tune_checkpoint: "pre-trained-models/ckpt-0"
  fine_tune_checkpoint_type: "detection"
...
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base:  .04
          total_steps: 20000
          warmup_learning_rate: .001
          warmup_steps: 500
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }

After finished training with the above options I updated the checkpoint file/type and momentum optimizer as so:

train_config: {
  fine_tune_checkpoint: "model-dir/ckpt-21"
  fine_tune_checkpoint_type: "full"
...
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base:  .04
          total_steps: 10000
          warmup_learning_rate: .02
          warmup_steps: 500
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }

I also tried not using the latest checkpoint as I thought this issue was caused by the training_loss being 0 at the end of the original run, but when I used checkpoint 16 (step 16000 out of 20000), the same issue occurred.

4. Expected behavior

I expect for the latest checkpoint to be loaded and then for the model to continue to train/run with nonzero learning-rate

5. Additional context

N/A.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Amazon Linux 2
• Mobile device name if the issue happens on a mobile device: N/A
• TensorFlow installed from (source or binary): Source
• TensorFlow version (use command below): 2.7.0
• Python version: 3.8.12
• Bazel version (if compiling from source): N/A
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: CUDA -- 11.2, CUDNN-- 8.1.1.33
• GPU model and memory: 4 Tesla V100 each 1600MiB