NVIDIA / NVIDIA/Megatron-LM

Add adapter-only checkpointing to Megatron-LM

Open
#7,270 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Parent initiative: #7267

## Work

Add adapter-only checkpoint save and load support to `megatron.training.checkpointing`.

## Done when

- Megatron-LM can save adapter weights without writing a second copy of the frozen base-model weights.
- Megatron-LM can load an adapter-only checkpoint onto the matching base model.
- The checkpoint contents remain compatible with the Megatron-Bridge PEFT workflow.
- Resuming PEFT training restores the adapter state required to continue training.
- Tests cover adapter-only save, load, and resume behavior.

Contributor guide

Open the contributing guide

Research direction

Start in megatron.training.checkpointing and trace the existing save, load, and resume paths for PEFT training. Compare the checkpoint expectations of the Megatron-Bridge PEFT workflow, then add tests for adapter-only save, load onto a matching base model, and resume; done means frozen base weights are not duplicated and adapter state is restored compatibly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.