Lightning-AI / Lightning-AI/pytorch-lightning

Change `optimizer` or `lr_scheduler` in resuming training without removing the `global_step` information

Open
#20,552 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lr scheduler optimizer question
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

Hello,

In case we want to resume training from a checkpoint, but for say change the `optimizer` class or the `lr_scheduler` class, it seems the `global_step` becomes 0. Is there a possibility to keep the global step information and still allow the change? This would greatly benefit if in case we are using a warmup + decay lr_scheduler, and on resume training, we want to change the number of decay steps. However with this change if the global step becomes 0, the model will do a warmup at resume, which is not intended.

### Pitch

_No response_

### Alternatives

_No response_

### Additional context

_No response_

cc @lantiga @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the checkpoint-resume path for optimizer and lr_scheduler state, focusing on where global_step is restored or reset. Done means changing either class while resuming preserves the checkpoint's global_step and avoids an unintended scheduler warmup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.