NVIDIA / NVIDIA/Megatron-LM

Consolidate PEFT support into Megatron-LM

Open
#7,267 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Goal

Move the PEFT support required by Megatron-Bridge into Megatron-LM so Bridge PEFT recipes can use `megatron.training` as the canonical training loop.

## Scope

- Move performant LoRA, canonical LoRA, DoRA linear adapters, and the `LoRALinear` wrapper into Megatron Core.
- Apply adapters and freeze the base model before model wrapping in the Megatron-LM training loop.
- Add adapter-only checkpoint save and load support to `megatron.training.checkpointing`.
- Migrate Megatron-Bridge PEFT recipes to the Megatron-LM training loop.

## Completion criteria

- Existing Megatron-Bridge PEFT APIs continue to work through the migration.
- Existing PEFT recipes retain their current behavior when run through the Megatron-LM training loop.
- Representative performant LoRA, canonical LoRA, and DoRA recipes preserve convergence and performance within agreed tolerances.
- Adapter-only checkpointing, adapter merging, and adapter export continue to work.

Contributor guide

Open the contributing guide

Research direction

Start by reading the Megatron-LM training loop and megatron.training.checkpointing, then compare the existing Megatron-Bridge PEFT APIs and recipes. Done means the PEFT implementations and migration preserve recipe behavior, adapter-only checkpointing, merging, export, convergence, and performance within agreed tolerances.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.