NVIDIA / NVIDIA/Megatron-LM

Apply PEFT adapters in the Megatron-LM training loop

Open
#7,269 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Parent initiative: #7267

## Work

Add a pre-wrap PEFT hook to the Megatron-LM training path. The hook must apply configured adapters and freeze the base model before distributed model wrapping.

## Done when

- `megatron.training` can apply the migrated PEFT adapters during model setup.
- Adapter application happens before DDP or FSDP wraps the model.
- Base-model parameters are frozen while adapter parameters remain trainable.
- The path supports performant LoRA, canonical LoRA, and DoRA.
- Tests verify adapter placement, parameter freezing, and ordering relative to model wrapping.

Contributor guide

Open the contributing guide

Research direction

Start at the megatron.training model-setup path and trace where DDP or FSDP wrapping occurs. Check how migrated PEFT adapters are configured, then verify adapter placement, base-parameter freezing, trainability, and ordering for performant LoRA, canonical LoRA, and DoRA with tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.