Move PEFT model components into Megatron Core
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Parent initiative: #7267
## Work
Move the reusable PEFT model components from Megatron-Bridge into a Megatron Core package such as `megatron.core.transformer.peft`.
This includes:
- Performant LoRA
- Canonical LoRA
- DoRA linear adapters
- The `LoRALinear` wrapper
## Done when
- Megatron Core owns the reusable adapter implementations.
- The components do not depend on Megatron-Bridge training or recipe code.
- Megatron-Bridge can import the Megatron Core implementations without maintaining duplicate copies.
- Unit tests cover the supported adapter types and their parameter behavior.
Contributor guide
Research direction
Start by locating the PEFT implementations and their unit tests in Megatron-Bridge, including Performant LoRA, Canonical LoRA, DoRA linear adapters, and the LoRALinear wrapper. Trace their dependencies on Bridge training or recipe code before choosing the Megatron Core package location. Done means Core owns the reusable implementations, Bridge imports them without duplicates, and tests cover supported adapter types and parameter behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100