modelscope / modelscope/ms-swift

Add fine-tuning support for LightOnOCR-1B-1025 model

Open
#6,490 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

more models
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Description

Describe the feature
Please add support for fine-tuning the [LightOnOCR-1B-1025](https://huggingface.co/lightonai/LightOnOCR-1B-1025) model.
This model is an end-to-end differentiable vision-language OCR model developed by LightOn AI, capable of performing OCR, layout analysis, and multilingual document understanding.
The model card mentions upcoming Transformers integration and support for LoRA/domain fine-tuning, but currently there’s no standard fine-tuning interface or example script available.

Fine-tuning support (either native or through transformers + peft integration) would enable developers to adapt this model for specific document domains — receipts, forms, scientific articles, invoices, etc.


Paste any useful information

Additional context
Fine-tuning support will allow users to:

  • Adapt OCR for domain-specific layouts or languages
  • Experiment with low-resource adaptation via LoRA
  • Easily share fine-tuned checkpoints on Hugging Face
  • Increase adoption of LightOnOCR across document AI research and production use-cases

If possible, please also include an example notebook or script demonstrating minimal fine-tuning workflow using the ModelScope or Hugging Face ecosystem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the LightOnOCR-1B-1025 model card and blog reference, then evaluate the suggested VisionEncoderDecoderModel integration and PEFT/LoRA adapter configuration. Done should mean a standard fine-tuning interface plus a minimal ModelScope or Hugging Face notebook or script demonstrating the workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.