Lightning-AI / Lightning-AI/pytorch-lightning

Please allow automatic optimization for multiple optimizers again.

Open
#20,094 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discussion feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

I'm suggesting allowing the old behavior to work again. While still giving users the option to use the new behavior if they set self.automatic_optimization=False. The old API was well designed and can allow for extremely simple implementations especially in situations where the training step is the same for each optimizer being used (i.e. no optimizer_idx if-statement).

### Pitch

The original purpose of Pytorch-Lightning was to simplify & eliminate the boiler plate in the pytorch training loop. But the new behavior is **much more complicated than even using base pytorch**. Since it requires extra bloat like `self.automatic_optimization=False`, `self.toggle_optimizer()`, `self.untoggle_optimizer()`, `self.optimizers()`, then using custom rewrites of well-known base pytorch APIs like `self.manual_backwards()`, in addition to reintroducing the boiler-plate that Pytorch-Lightning was made to remove.

As a matter of fact **in the simplest case it adds 12 additional lines of bloat...**

### Alternatives

_No response_

### Additional context

Could you at least consider collecting user feedback before you remove useful features like this in the future?

cc @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the automatic_optimization and multiple-optimizer behavior described in the issue, including the manual APIs listed: toggle_optimizer(), untoggle_optimizer(), optimizers(), and manual_backwards(). Trace the training loop entry points and existing feedback or tests for optimizer handling. Done would require a decided API design that restores the requested automatic behavior while preserving the opt-out path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.