deepinsight / deepinsight/insightface

Train recognition: the learning rate scheduler

Open
#981 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

I got confused on the learning rate scheduler in [recognition/train.py](https://github.com/deepinsight/insightface/blob/4a4b8d03fec981912fdef5b3232a37a827cbeed6/recognition/train.py#L291-L303)

Relevant code is:
```
def _batch_callback(param):
#global global_step
global_step[0]+=1
mbatch = global_step[0]
for step in lr_steps:
if mbatch==step:
opt.lr *= 0.1
print('lr change to', opt.lr)
break

_cb(param)
if mbatch%1000==0:
print('lr-batch-epoch:',opt.lr,param.nbatch,param.epoch)
```

So what's the difference between `mbatch`, `nbatch` and `global_step` here?

For me I trained with the default configurations
(where `lr_steps='100000,160000,220000`) with batch-size=110 (the maximal bs I can have, due to the GPUs Mem limitation) on a 4-GPUs machine.

From my log the learning rate changes at a very strange batch index:
```
lr change to 0.010000000000000002
lr-batch-epoch: 0.010000000000000002 7368 7
```

In my understanding the lr should decrease for the first time when `mbatch=100K`.
But in my practice it decreases at

Can someone explain the mechanism here?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.