huggingface / huggingface/peft

[Proposal] Add CLoRA a new PEFT method

Open
#3,542 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### Feature request

Dear PEFT team,

I would like to propose adding our published method, [Controlled LoRA (CLoRA)](https://aclanthology.org/2025.acl-long.940/), from ACL 2025, to the PEFT library. The official reference implementation is available [here](https://github.com/sutakori/CLoRA).

CLoRA is a subspace regularization method built on top of the standard LoRA structure. It is designed to mitigate catastrophic forgetting during continued fine-tuning while preserving sufficient model capacity for learning the new task.

Specifically, CLoRA regularizes the LoRA update by constraining selected directions associated with the null space of the update matrix. The regularization is applied during training without changing the standard LoRA adapter structure used for inference. Our experiments on both one-stage fine-tuning and continual learning settings show that CLoRA achieves a better balance between downstream-task adaptation and retention of the model's previous capabilities.

Since CLoRA operates on the existing LoRA structure, I expect that it could potentially be integrated as a LoRA variant or as an auxiliary regularization option within the current LoRA tuner. I would appreciate the maintainers' guidance on the preferred integration approach.

### Your contribution

As the first author of the CLoRA paper and the maintainer of its official implementation, I would be happy to implement the core integration of CLoRA and submit a PR if the method is considered within the scope of the PEFT library.

Contributor guide

Open the contributing guide

Research direction

Start by reading PEFT's current LoRA tuner and the official CLoRA implementation linked in the issue, then compare how the proposed regularization fits the existing adapter structure. Done means reaching agreement on the integration approach and implementing CLoRA as either a LoRA variant or an auxiliary regularization option, with the expected training and retention behavior demonstrated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.