Lightning-AI / Lightning-AI/pytorch-lightning

Strict warning about using optimizers manually would be needed in the docs

Open Beginner friendly
#17,281 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

docs help wanted
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### 📚 Documentation

```
# Correct
self.optimizers()[0].zero_grad()
self.optimizers()[1].zero_grad()
self.manual_backward(loss)
self.optimizers()[0].step()
self.optimizers()[1].step()

# Wrong
self.optimizer0.zero_grad()
self.optimizer1.zero_grad()
self.manual_backward(loss)
self.optimizer0.step()
self.optimizer1.step()
```
I've struggled with the issue of ModelCheckpoint module not saving checkpoint files to the designated directory. And I just found out from [this issue](https://github.com/Lightning-AI/lightning/issues/13752) that the difference of code above can cause global_step not increasing because global_step refers to the sum of all optimizer.step() calls, which leads to the situation that self._last_global_step_saved == trainer.global_step in _should_skip_saving_checkpoint() function is always True. This is very frustrating for newcomers like me because it's hardly found any relations between the coding style of accessing optimizers and global step. So I suggest global_step be more intuitive (for example, the number of training_step ends), or put some explanation in the [docs](https://lightning.ai/docs/pytorch/latest/common/optimization.html) about why it is necessary accessing optimizers with self.optimizers() function, not directly.

cc @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the manual optimization section at https://lightning.ai/docs/pytorch/latest/common/optimization.html and review how optimizer access, global_step, and checkpoint saving are described. Update the documentation to explain why self.optimizers() should be used instead of direct optimizer attributes, and make the relationship to global_step and ModelCheckpoint clear using the reported examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.