Lightning-AI / Lightning-AI/pytorch-lightning
Strict warning about using optimizers manually would be needed in the docs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### 📚 Documentation
```
# Correct
self.optimizers()[0].zero_grad()
self.optimizers()[1].zero_grad()
self.manual_backward(loss)
self.optimizers()[0].step()
self.optimizers()[1].step()
# Wrong
self.optimizer0.zero_grad()
self.optimizer1.zero_grad()
self.manual_backward(loss)
self.optimizer0.step()
self.optimizer1.step()
```
I've struggled with the issue of ModelCheckpoint module not saving checkpoint files to the designated directory. And I just found out from [this issue](https://github.com/Lightning-AI/lightning/issues/13752) that the difference of code above can cause global_step not increasing because global_step refers to the sum of all optimizer.step() calls, which leads to the situation that self._last_global_step_saved == trainer.global_step in _should_skip_saving_checkpoint() function is always True. This is very frustrating for newcomers like me because it's hardly found any relations between the coding style of accessing optimizers and global step. So I suggest global_step be more intuitive (for example, the number of training_step ends), or put some explanation in the [docs](https://lightning.ai/docs/pytorch/latest/common/optimization.html) about why it is necessary accessing optimizers with self.optimizers() function, not directly.
cc @borda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the manual optimization section at https://lightning.ai/docs/pytorch/latest/common/optimization.html and review how optimizer access, global_step, and checkpoint saving are described. Update the documentation to explain why self.optimizers() should be used instead of direct optimizer attributes, and make the relationship to global_step and ModelCheckpoint clear using the reported examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100