tensorflow / tensorflow/models

Default learning rate in TF2 SSD MobileNet V2 config file is way too high. Is it a typo?

Open
#10,509 4 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Feb 23, 2022.

models:research:odapi type:bug
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

The learning rate set in the TF2 SSD MobileNet V2 config file is 10x higher than that of the other SSD MobileNet models. This causes loss during training to get extremely high. Is it a typo?

The default ssd_mobilenet_v2_320x320_coco17_tpu-8.config configuration has this for the learning rate:

  optimizer {
    momentum_optimizer: {
      learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base: .8
          total_steps: 50000
          warmup_learning_rate: 0.13333
          warmup_steps: 2000
        }
      }
      momentum_optimizer_value: 0.9

Meanwhile, the FPNLite version ( ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.config ) has this:

  optimizer {
    momentum_optimizer: {
      learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base: .08
          total_steps: 50000
          warmup_learning_rate: .026666
          warmup_steps: 1000
        }
      }
      momentum_optimizer_value: 0.9

When I train with the default values in the ssd_mobilenet_v2_320x320_coco17_tpu-8.config file, the huge learning rate throws training way off. When I change the values from .8 to .08 and .13333 to .013333, training works much better. I think whoever wrote the config file missed a decimal point.

Loss graph BEFORE changing learning rate values (the loss is way higher)
image

Loss graph AFTER changing learning rate values
image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.