NVIDIA / NVIDIA/Megatron-LM

Save and load train_state.pt in checkpoints

Open
#6,004 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Parent initiative: #2319

Depends on #3555 for the `TrainState` runtime contract.

## Context

`TrainState` will be the runtime source of truth for mutable training progress. This task covers persisting and restoring that state through Megatron-LM checkpoints while preserving compatibility with older checkpoints.

## Scope

- Save the active `TrainState` as `train_state.pt` in the checkpoint iteration directory.
- Load `train_state.pt` into the active `TrainState` when the sidecar is present.
- Make the loaded state available consistently across distributed ranks.
- Prefer `train_state.pt` when both the sidecar and legacy progress fields are present.
- When `train_state.pt` is absent, construct `TrainState` from the legacy checkpoint fields/`state_dict["args"]`.
- Verify that the sidecar and legacy fields contain equivalent values while both are written.
- Leave the distributed checkpoint representation for model, optimizer, scheduler, RNG, and rerun state unchanged.

## Completion criteria

- New Megatron-LM checkpoints contain `train_state.pt`.
- Synchronous and asynchronous checkpoint saves write the same training-state metadata.
- Saving and loading a checkpoint recovers every `TrainState` field.
- New Megatron-LM code reconstructs `TrainState` from older checkpoints without the sidecar.
- Tests cover round trips, legacy checkpoint format, sidecar precedence, and agreement between the two representations.

## Out of scope

- Defining the `TrainState` schema or lifecycle.
- Migrating training, evaluation, or logging code away from mutable fields in `args`.
- Saving or loading `PretrainConfigContainer`.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the checkpoint save and load paths and reading the TrainState runtime contract from #3555. Implement and test train_state.pt handling in the checkpoint iteration directory, including synchronous and asynchronous saves, distributed rank consistency, legacy reconstruction, precedence, and equivalent sidecar and legacy values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.