Apply PEFT adapters in the Megatron-LM training loop
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Parent initiative: #7267
## Work
Add a pre-wrap PEFT hook to the Megatron-LM training path. The hook must apply configured adapters and freeze the base model before distributed model wrapping.
## Done when
- `megatron.training` can apply the migrated PEFT adapters during model setup.
- Adapter application happens before DDP or FSDP wraps the model.
- Base-model parameters are frozen while adapter parameters remain trainable.
- The path supports performant LoRA, canonical LoRA, and DoRA.
- Tests verify adapter placement, parameter freezing, and ordering relative to model wrapping.
Contributor guide
Research direction
Start at the megatron.training model-setup path and trace where DDP or FSDP wrapping occurs. Check how migrated PEFT adapters are configured, then verify adapter placement, base-parameter freezing, trainability, and ordering for performant LoRA, canonical LoRA, and DoRA with tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100