facebookresearch / facebookresearch/detectron2
Change optimizer features between training phases
- Dominant language
- Python
- Stars
- 34.7k
- Forks
- 7.9k
- PR merge metrics
- No merged PRs in 30d
Description
Hi all, thank for the great implementation of Detectron2 !
I was wondering if it was possible to change the number of resnet trained layers between two consecutive training like it was possible with this implementation in Tensorflow.
https://github.com/crowdAI/crowdai-mapping-challenge-mask-rcnn/blob/master/Training.ipynb
```
# *** This training schedule is an example. Update to your needs ***
# Training - Stage 1
print("Training network heads")
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=40,
layers='heads')
# Training - Stage 2
# Finetune layers from ResNet stage 4 and up
print("Fine tune Resnet stage 4 and up")
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=120,
layers='4+')
# Training - Stage 3
# Fine tune all layers
print("Fine tune all layers")
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE / 10,
epochs=160,
layers='all')
```
I am just training Mask RCNN over basic aerial imagery and I tried to define new config with just adding `cfg2.MODEL.BACKBONE.FREEZE_AT=1` and `cfg2.MODEL.BACKBONE.FREEZE_AT=0` to them to get 2 new defaultrainer and continue consecutive training with `trainer2.resume_or_load(resume=True)`.
but I still get the error `ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group`.
```
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()
trainer2 = DefaultTrainer(cfg2)
trainer2.resume_or_load(resume=True)
trainer2.train()
trainer3 = DefaultTrainer(cfg3)
trainer3.resume_or_load(resume=True)
trainer3.train()
```
It seems the optimizer features loaded from the checkpoint do not match the optimizer features I want to change for the two trainer and as a matter of fact the code is returning an error.
Are there any way to overcome this error and somehow force the code to update the optimizer parameters when continuing training ?
Thank in advance for anybody that could help me 😊
Contributor guide
Research direction
Start with DefaultTrainer and the resume_or_load workflow described in the issue, then reproduce the failure by changing MODEL.BACKBONE.FREEZE_AT between consecutive training runs. Trace how the checkpoint restores optimizer state and determine what behavior is needed when optimizer parameter groups change. Done means staged training can resume with the requested optimizer parameters without the size-mismatch error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100