Lightning-AI / Lightning-AI/pytorch-lightning

Easier change optimizer/learning rate instead of reading state_dict from checkpoint

Open
#4,044 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature help wanted
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature
Easier change optimizer / learning rate without recoding

### Motivation

Usually when we want to change the optimizer from one to another(for example,using Adam to fast init training and turn to SGD at the end) in PyTorch, we may have to load state_dict of the model and change then change the optimizer in the code.

In PytorchLightning, model state_dict, state of all optimizers, state of all learning rate schedulers etc. are automately saved in the lightning checkpoint. It seems that we will still have to load state_dict of the model from the checkpoint. (Similar issues also happen when we want to change the learning rate).

Is it possible that we are able to directly change the code and use a new optimizer/ new learning rate instead of recoding and reading the state_dict from checkpoint?

### Pitch

Directly change the code and then a new optimizer/ new learning rate could be used.

### Alternatives

### Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by tracing how model, optimizer, and learning-rate-scheduler state is restored from Lightning checkpoints, then define the expected behavior when code selects a different optimizer or learning rate. Done means a resumed run can apply those changes without manually reading or rewriting the model state_dict, with coverage for both optimizer replacement and learning-rate changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.