Lightning-AI / Lightning-AI/pytorch-lightning
Easier change optimizer/learning rate instead of reading state_dict from checkpoint
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
Easier change optimizer / learning rate without recoding
### Motivation
Usually when we want to change the optimizer from one to another(for example,using Adam to fast init training and turn to SGD at the end) in PyTorch, we may have to load state_dict of the model and change then change the optimizer in the code.
In PytorchLightning, model state_dict, state of all optimizers, state of all learning rate schedulers etc. are automately saved in the lightning checkpoint. It seems that we will still have to load state_dict of the model from the checkpoint. (Similar issues also happen when we want to change the learning rate).
Is it possible that we are able to directly change the code and use a new optimizer/ new learning rate instead of recoding and reading the state_dict from checkpoint?
### Pitch
Directly change the code and then a new optimizer/ new learning rate could be used.
### Alternatives
### Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing how model, optimizer, and learning-rate-scheduler state is restored from Lightning checkpoints, then define the expected behavior when code selects a different optimizer or learning rate. Done means a resumed run can apply those changes without manually reading or rewriting the model state_dict, with coverage for both optimizer replacement and learning-rate changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100