Lightning-AI / Lightning-AI/lightning-thunder

Workaround for Adam.step's lazy initialization of its inner state

Open
#1,057 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

#### The next comment explains what this issue aims at

When `thunder.jit` trace `optimizer._init_group` before this initialization, calls of the jitted `optimizer.step` will all reset the inner state. To make it work correctly, the following conditions are necessary:
* The obtained trace does not initialize the inner state.
* The inner state gets initialized independently from trace execution.

To get consistent `optimizer.state_dict()` requires another condition:
* The actual tensors (not their proxies) are accessible on initialization.

The simplest way would be to actually call `optimizer._init_group` at the early stage to guarantee the inner state is initialized before tracing it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with thunder.jit tracing around optimizer._init_group and optimizer.step, then inspect how optimizer.state_dict() observes the inner state. The work is done when tracing no longer resets that state and state_dict() exposes the actual initialized tensors rather than proxies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.