google-research / google-research/t5x

How to initialize a train state from a PyTree of parameters?

Open
#1,397 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3k
Forks
338
PR merge metrics
No merged PRs in 30d

Description

Hi,

I am just wondering is there an efficient and straightforward function/method to initialize a train state from a PyTree of model parameters? Currently I can find `Checkpointer.restore` to restore a train state from checkpoints `TrainStateInitializer.from_scratch` to intialize a train state from scratch.

Currently I am trying to load a dictionary of model parameters (without training steps and optimizer states, etc.) and would like to do continual training on these parameters.

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with TrainStateInitializer.from_scratch and Checkpointer.restore to understand the existing initialization and restoration paths. Determine whether the requested PyTree-only initialization is supported or needs a new entry point, with completion defined as continuing training from supplied parameters without checkpoint-only state.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.