facebookresearch / facebookresearch/detectron2

Change optimizer features between training phases

Open
#4,257 0 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

Hi all, thank for the great implementation of Detectron2 !

I was wondering if it was possible to change the number of resnet trained layers between two consecutive training like it was possible with this implementation in Tensorflow.
https://github.com/crowdAI/crowdai-mapping-challenge-mask-rcnn/blob/master/Training.ipynb

```
# *** This training schedule is an example. Update to your needs ***

# Training - Stage 1
print("Training network heads")
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=40,
layers='heads')

# Training - Stage 2
# Finetune layers from ResNet stage 4 and up
print("Fine tune Resnet stage 4 and up")
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=120,
layers='4+')

# Training - Stage 3
# Fine tune all layers
print("Fine tune all layers")
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE / 10,
epochs=160,
layers='all')
```

I am just training Mask RCNN over basic aerial imagery and I tried to define new config with just adding `cfg2.MODEL.BACKBONE.FREEZE_AT=1` and `cfg2.MODEL.BACKBONE.FREEZE_AT=0` to them to get 2 new defaultrainer and continue consecutive training with `trainer2.resume_or_load(resume=True)`.
but I still get the error `ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group`.

```
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

trainer2 = DefaultTrainer(cfg2)
trainer2.resume_or_load(resume=True)
trainer2.train()

trainer3 = DefaultTrainer(cfg3)
trainer3.resume_or_load(resume=True)
trainer3.train()

```

It seems the optimizer features loaded from the checkpoint do not match the optimizer features I want to change for the two trainer and as a matter of fact the code is returning an error.

Are there any way to overcome this error and somehow force the code to update the optimizer parameters when continuing training ?

Thank in advance for anybody that could help me 😊

Contributor guide

Open the contributing guide

Research direction

Start with DefaultTrainer and the resume_or_load workflow described in the issue, then reproduce the failure by changing MODEL.BACKBONE.FREEZE_AT between consecutive training runs. Trace how the checkpoint restores optimizer state and determine what behavior is needed when optimizer parameter groups change. Done means staged training can resume with the requested optimizer parameters without the size-mismatch error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.