huggingface / huggingface/peft
Proposal: add FineGates structured-sparsity PEFT method
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 59
Description
### Feature request
We would like to discuss adding **FineGates**, a parameter-efficient fine-tuning method based on structured sparsification, to PEFT.
FineGates adapts frozen language models by learning stochastic gates over selected rows and columns. Unlike adapter-based approaches that add trainable modules, the resulting structured sparsity can reduce inference cost as well as training overhead. In our experiments, the method removes 20--40% of model parameters without significant accuracy loss while using a small number of trainable gate parameters.
Stable references:
- AISTATS 2026 paper: [Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity](https://openreview.net/forum?id=jU4ERfrjpH)
- arXiv: https://arxiv.org/abs/2602.09169
- Implementation repository: https://github.com/jsvir/FineGates
### Motivation
FineGates offers a different efficiency tradeoff from LoRA-style methods: it does not introduce extra adapter weights at inference time, and its structured pruning can reduce model size and inference latency after fine-tuning. We think this makes it a useful complement to the existing PEFT methods, especially for deployment-constrained settings.
### Proposed integration
If the maintainers consider the method a good fit, we can prepare a PR following the PEFT contribution guide. The initial integration would include:
- a new `PeftType` and tuner package;
- configuration for the structured gates;
- training and export/compression behavior;
- tests, documentation, and a runnable example;
- benchmark settings for comparison with existing methods.
### Our contribution
We are authors of the paper and would be happy to prepare the integration PR and support maintenance. We would appreciate feedback on whether this belongs as a native PEFT tuner and on the preferred interface for exporting the compressed model.
Contributor guide
Assessment
This issue has not been assessed yet.