Consolidate PEFT support into Megatron-LM
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
## Goal
Move the PEFT support required by Megatron-Bridge into Megatron-LM so Bridge PEFT recipes can use `megatron.training` as the canonical training loop.
## Scope
- Move performant LoRA, canonical LoRA, DoRA linear adapters, and the `LoRALinear` wrapper into Megatron Core.
- Apply adapters and freeze the base model before model wrapping in the Megatron-LM training loop.
- Add adapter-only checkpoint save and load support to `megatron.training.checkpointing`.
- Migrate Megatron-Bridge PEFT recipes to the Megatron-LM training loop.
## Completion criteria
- Existing Megatron-Bridge PEFT APIs continue to work through the migration.
- Existing PEFT recipes retain their current behavior when run through the Megatron-LM training loop.
- Representative performant LoRA, canonical LoRA, and DoRA recipes preserve convergence and performance within agreed tolerances.
- Adapter-only checkpointing, adapter merging, and adapter export continue to work.
Contributor guide
Research direction
Start by reading the Megatron-LM training loop and megatron.training.checkpointing, then compare the existing Megatron-Bridge PEFT APIs and recipes. Done means the PEFT implementations and migration preserve recipe behavior, adapter-only checkpointing, merging, export, convergence, and performance within agreed tolerances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100