huggingface / huggingface/peft

Proposal: support ShadowPEFT method in PEFT

Open
#3,184 6 comments 0 reactions 0 assignees View on GitHub
wip
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### Feature request

Dear PEFT team,

I am a big fan of the peft library. Thanks for your contributions to making this project so easy to use.

Recently, we proposed a new PEFT method -- **ShadowPEFT** ([ArXiv](https://arxiv.org/abs/2604.19254) | [HuggingFace](https://huggingface.co/collections/shadow-llm/shadow-peft-models)).

Unlike LoRAs that inject trainable parameters to the base's weights (weight-level), ShadowPEFT centralizes trainable parameters in a shadow network (architecturally similar to the backbone) and injects it at each Transformer layer (layer-level).

The shadow network operates in parallel with the base model, delivering learned corrections to each decoder layer. Because the shadow module is architecturally decoupled from the base, it can be trained, stored, and deployed independently, benefiting edge computing scenarios and edge-cloud collaboration.

We followed PEFT's API to implement ShadowPEFT in [GitHub:ShadowLLM/shadow-peft](https://github.com/ShadowLLM/shadow-peft). We would be thrilled to see ShadowPEFT considered for integration into the official PEFT library.

Best,
Xianming

### Your contribution

- integration test

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.