Align Megatron-LM and Megatron-Bridge checkpoint metadata
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
## Goal
Align checkpoint metadata between Megatron-LM and Megatron-Bridge while keeping the existing distributed checkpoint representation for model and optimizer state.
## Current state
The two implementations already store model weights, optimizer state, scheduler state, RNG state, and rerun state in the same format. Configuration and training progress differ:
- Megatron-Bridge writes configuration to `run_config.yaml` and mutable training progress to `train_state.pt`.
- Megatron-LM stores configuration in `state_dict["args"]`. It stores training progress in `args` and top-level checkpoint fields.
- Megatron-Bridge can load legacy Megatron-LM checkpoints. Megatron-LM does not yet load the Megatron-Bridge metadata format.
## Scope
- Add YAML serialization to `PretrainConfigContainer` and save it as `run_config.yaml`.
- Save mutable training progress as a `TrainState` in `train_state.pt`.
- Load `run_config.yaml` and `train_state.pt` when they are present.
- Fall back to `state_dict["args"]` and the existing progress fields when loading older Megatron-LM checkpoints.
- Keep the legacy metadata while older Megatron-LM readers still need it.
## Completion criteria
- New Megatron-LM code can load existing Megatron-LM checkpoints.
- Megatron-LM can load checkpoints containing Megatron-Bridge-style configuration and training-state metadata.
- Save and load tests recover the expected `PretrainConfigContainer` and `TrainState`.
- The distributed checkpoint representation for model, optimizer, scheduler, RNG, and rerun state does not change.
## Out of scope
- Changing the distributed checkpoint representation for model or optimizer state.
- Removing `state_dict["args"]`. That can be considered after the compatibility period.
Contributor guide
Assessment
This issue has not been assessed yet.