modelscope / modelscope/ms-swift
Add fine-tuning support for LightOnOCR-1B-1025 model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Description
Describe the feature
Please add support for fine-tuning the [LightOnOCR-1B-1025](https://huggingface.co/lightonai/LightOnOCR-1B-1025) model.
This model is an end-to-end differentiable vision-language OCR model developed by LightOn AI, capable of performing OCR, layout analysis, and multilingual document understanding.
The model card mentions upcoming Transformers integration and support for LoRA/domain fine-tuning, but currently there’s no standard fine-tuning interface or example script available.
Fine-tuning support (either native or through transformers + peft integration) would enable developers to adapt this model for specific document domains — receipts, forms, scientific articles, invoices, etc.
Paste any useful information
-
Model card: https://huggingface.co/lightonai/LightOnOCR-1B-1025
-
Blog reference: https://huggingface.co/blog/lightonai/lightonocr
-
Suggested integration targets:
transformers.VisionEncoderDecoderModelsubclass- PEFT/LoRA adapter configuration for efficient fine-tuning
Additional context
Fine-tuning support will allow users to:
- Adapt OCR for domain-specific layouts or languages
- Experiment with low-resource adaptation via LoRA
- Easily share fine-tuned checkpoints on Hugging Face
- Increase adoption of LightOnOCR across document AI research and production use-cases
If possible, please also include an example notebook or script demonstrating minimal fine-tuning workflow using the ModelScope or Hugging Face ecosystem.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the LightOnOCR-1B-1025 model card and blog reference, then evaluate the suggested VisionEncoderDecoderModel integration and PEFT/LoRA adapter configuration. Done should mean a standard fine-tuning interface plus a minimal ModelScope or Hugging Face notebook or script demonstrating the workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100