NVIDIA / NVIDIA/Megatron-LM

Move PEFT model components into Megatron Core

Open
#7,271 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Parent initiative: #7267

## Work

Move the reusable PEFT model components from Megatron-Bridge into a Megatron Core package such as `megatron.core.transformer.peft`.

This includes:

- Performant LoRA
- Canonical LoRA
- DoRA linear adapters
- The `LoRALinear` wrapper

## Done when

- Megatron Core owns the reusable adapter implementations.
- The components do not depend on Megatron-Bridge training or recipe code.
- Megatron-Bridge can import the Megatron Core implementations without maintaining duplicate copies.
- Unit tests cover the supported adapter types and their parameter behavior.

Contributor guide

Open the contributing guide

Research direction

Start by locating the PEFT implementations and their unit tests in Megatron-Bridge, including Performant LoRA, Canonical LoRA, DoRA linear adapters, and the LoRALinear wrapper. Trace their dependencies on Bridge training or recipe code before choosing the Megatron Core package location. Done means Core owns the reusable implementations, Bridge imports them without duplicates, and tests cover supported adapter types and parameter behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.