Lightning-AI / Lightning-AI/lightning-thunder
Workaround for Adam.step's lazy initialization of its inner state
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
#### The next comment explains what this issue aims at
When `thunder.jit` trace `optimizer._init_group` before this initialization, calls of the jitted `optimizer.step` will all reset the inner state. To make it work correctly, the following conditions are necessary:
* The obtained trace does not initialize the inner state.
* The inner state gets initialized independently from trace execution.
To get consistent `optimizer.state_dict()` requires another condition:
* The actual tensors (not their proxies) are accessible on initialization.
The simplest way would be to actually call `optimizer._init_group` at the early stage to guarantee the inner state is initialized before tracing it.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with thunder.jit tracing around optimizer._init_group and optimizer.step, then inspect how optimizer.state_dict() observes the inner state. The work is done when tracing no longer resets that state and state_dict() exposes the actual initialized tensors rather than proxies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100