huggingface / huggingface/peft

Proposal: Benchmarking on different PEFT techniques

Open
#3,363 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

I recently read this Hugging Face blog: [**"Beyond LoRA: Can You Beat the Most Popular Fine-Tuning Technique?"**
](https://huggingface.co/blog/peft-beyond-lora)
The blog concludes that, depending on the downstream task, PEFT techniques other than LoRA can outperform it. Inspired by this benchmark, I'd like to contribute by evaluating different PEFT methods on **Table Recognition** and **OCR** tasks using datasets available on Hugging Face.

I plan to run a comprehensive set of PEFT experiments on OCR-focused vision-language models such as:

* `zai-org/GLM-OCR`
* `PaddlePaddle/PaddleOCR-VL-1.6`
* and other relevant OCR/Table Recognition models.

Example dataset: [apoidea/fintabnet-html](https://huggingface.co/datasets/apoidea/fintabnet-html)

The goal is to benchmark different PEFT methods across these tasks and compare their effectiveness in terms of accuracy, efficiency, and resource usage.

If this sounds like a valuable addition to the benchmark, I'd be happy to take ownership of this task. Please assign it to me if you think it would be a good fit.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Hugging Face blog, the GLM-OCR and PaddleOCR-VL-1.6 model resources, and the apoidea/fintabnet-html dataset. Define the benchmark scope and reproducible experiments for OCR and table recognition, then compare PEFT methods by accuracy, efficiency, and resource usage; completion means the results support a documented comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.