huggingface / huggingface/peft
[Proposal] Add CLoRA a new PEFT method
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 59
Description
### Feature request
Dear PEFT team,
I would like to propose adding our published method, [Controlled LoRA (CLoRA)](https://aclanthology.org/2025.acl-long.940/), from ACL 2025, to the PEFT library. The official reference implementation is available [here](https://github.com/sutakori/CLoRA).
CLoRA is a subspace regularization method built on top of the standard LoRA structure. It is designed to mitigate catastrophic forgetting during continued fine-tuning while preserving sufficient model capacity for learning the new task.
Specifically, CLoRA regularizes the LoRA update by constraining selected directions associated with the null space of the update matrix. The regularization is applied during training without changing the standard LoRA adapter structure used for inference. Our experiments on both one-stage fine-tuning and continual learning settings show that CLoRA achieves a better balance between downstream-task adaptation and retention of the model's previous capabilities.
Since CLoRA operates on the existing LoRA structure, I expect that it could potentially be integrated as a LoRA variant or as an auxiliary regularization option within the current LoRA tuner. I would appreciate the maintainers' guidance on the preferred integration approach.
### Your contribution
As the first author of the CLoRA paper and the maintainer of its official implementation, I would be happy to implement the core integration of CLoRA and submit a PR if the method is considered within the scope of the PEFT library.
Contributor guide
Research direction
Start by reading PEFT's current LoRA tuner and the official CLoRA implementation linked in the issue, then compare how the proposed regularization fits the existing adapter structure. Done means reaching agreement on the integration approach and implementing CLoRA as either a LoRA variant or an auxiliary regularization option, with the expected training and retention behavior demonstrated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100