google-research / google-research/t5x
How to initialize a train state from a PyTree of parameters?
- Dominant language
- Python
- Stars
- 3k
- Forks
- 338
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am just wondering is there an efficient and straightforward function/method to initialize a train state from a PyTree of model parameters? Currently I can find `Checkpointer.restore` to restore a train state from checkpoints `TrainStateInitializer.from_scratch` to intialize a train state from scratch.
Currently I am trying to load a dictionary of model parameters (without training steps and optimizer states, etc.) and would like to do continual training on these parameters.
Thanks!
Contributor guide
Research direction
Start with TrainStateInitializer.from_scratch and Checkpointer.restore to understand the existing initialization and restoration paths. Determine whether the requested PyTree-only initialization is supported or needs a new entry point, with completion defined as continuing training from supplied parameters without checkpoint-only state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100