NVIDIA / NVIDIA/Megatron-LM

Align Megatron-LM and Megatron-Bridge checkpoint metadata

Open
#2,319 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Goal

Align checkpoint metadata between Megatron-LM and Megatron-Bridge while keeping the existing distributed checkpoint representation for model and optimizer state.

## Current state

The two implementations already store model weights, optimizer state, scheduler state, RNG state, and rerun state in the same format. Configuration and training progress differ:

- Megatron-Bridge writes configuration to `run_config.yaml` and mutable training progress to `train_state.pt`.
- Megatron-LM stores configuration in `state_dict["args"]`. It stores training progress in `args` and top-level checkpoint fields.
- Megatron-Bridge can load legacy Megatron-LM checkpoints. Megatron-LM does not yet load the Megatron-Bridge metadata format.

## Scope

- Add YAML serialization to `PretrainConfigContainer` and save it as `run_config.yaml`.
- Save mutable training progress as a `TrainState` in `train_state.pt`.
- Load `run_config.yaml` and `train_state.pt` when they are present.
- Fall back to `state_dict["args"]` and the existing progress fields when loading older Megatron-LM checkpoints.
- Keep the legacy metadata while older Megatron-LM readers still need it.

## Completion criteria

- New Megatron-LM code can load existing Megatron-LM checkpoints.
- Megatron-LM can load checkpoints containing Megatron-Bridge-style configuration and training-state metadata.
- Save and load tests recover the expected `PretrainConfigContainer` and `TrainState`.
- The distributed checkpoint representation for model, optimizer, scheduler, RNG, and rerun state does not change.

## Out of scope

- Changing the distributed checkpoint representation for model or optimizer state.
- Removing `state_dict["args"]`. That can be considered after the compatibility period.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.