huggingface / huggingface/peft

Proposal: add FineGates structured-sparsity PEFT method

Open
#3,292 8 comments 0 reactions 0 assignees View on GitHub
wip
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### Feature request

We would like to discuss adding **FineGates**, a parameter-efficient fine-tuning method based on structured sparsification, to PEFT.

FineGates adapts frozen language models by learning stochastic gates over selected rows and columns. Unlike adapter-based approaches that add trainable modules, the resulting structured sparsity can reduce inference cost as well as training overhead. In our experiments, the method removes 20--40% of model parameters without significant accuracy loss while using a small number of trainable gate parameters.

Stable references:
- AISTATS 2026 paper: [Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity](https://openreview.net/forum?id=jU4ERfrjpH)
- arXiv: https://arxiv.org/abs/2602.09169
- Implementation repository: https://github.com/jsvir/FineGates

### Motivation

FineGates offers a different efficiency tradeoff from LoRA-style methods: it does not introduce extra adapter weights at inference time, and its structured pruning can reduce model size and inference latency after fine-tuning. We think this makes it a useful complement to the existing PEFT methods, especially for deployment-constrained settings.

### Proposed integration

If the maintainers consider the method a good fit, we can prepare a PR following the PEFT contribution guide. The initial integration would include:
- a new `PeftType` and tuner package;
- configuration for the structured gates;
- training and export/compression behavior;
- tests, documentation, and a runnable example;
- benchmark settings for comparison with existing methods.

### Our contribution

We are authors of the paper and would be happy to prepare the integration PR and support maintenance. We would appreciate feedback on whether this belongs as a native PEFT tuner and on the preferred interface for exporting the compressed model.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.