Add adapter-only checkpointing to Megatron-LM
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Parent initiative: #7267
## Work
Add adapter-only checkpoint save and load support to `megatron.training.checkpointing`.
## Done when
- Megatron-LM can save adapter weights without writing a second copy of the frozen base-model weights.
- Megatron-LM can load an adapter-only checkpoint onto the matching base model.
- The checkpoint contents remain compatible with the Megatron-Bridge PEFT workflow.
- Resuming PEFT training restores the adapter state required to continue training.
- Tests cover adapter-only save, load, and resume behavior.
Contributor guide
Research direction
Start in megatron.training.checkpointing and trace the existing save, load, and resume paths for PEFT training. Compare the checkpoint expectations of the Megatron-Bridge PEFT workflow, then add tests for adapter-only save, load onto a matching base model, and resume; done means frozen base weights are not duplicated and adapter state is restored compatibly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100