docling-project / docling-project/docling

Unlimited-OCR: MLX conversion for Apple silicon

Open
#3,943 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

Split out of #3679, where @geoHeil asked whether Unlimited-OCR could be converted to MLX and published
to the Hub so that docling keeps its "most models also run on Apple silicon" property.

Keeping it separate so the parsing work in #3679 stays reviewable on its own — that change is
API-first and does not depend on this one.

### What would be needed

- an MLX conversion of [`baidu/Unlimited-OCR`](https://huggingface.co/baidu/Unlimited-OCR) (MIT
licensed, so redistribution of a converted copy is permitted) published to the Hub;
- an `engine_overrides` entry for `VlmEngineType.MLX` on the `unlimited_ocr` preset pointing at that
repo id;
- a check that the model still emits its grounding annotations under MLX. This matters more than usual
here: the model has no chat template and is very sensitive to prompt wording — the vendor prompt
`\n<|grounding|>OCR this image.` works, while generic "convert to markdown" phrasings return an
empty completion with `finish_reason: stop` and no error. A conversion that silently changes tokenizer
behaviour would look like an empty page rather than a failure.
- the same for `skip_special_tokens`: the grounding markers are special tokens, and on the
OpenAI-compatible path they are stripped unless `skip_special_tokens=False` is passed.

### Status

We have Apple silicon available and intend to look at this, but not as part of the parsing PR. Anyone
who gets there first is welcome — please comment here so the work is not duplicated.

Contributor guide

Open the contributing guide

Research direction

Start with the unlimited_ocr preset and its engine_overrides entry for VlmEngineType.MLX, then inspect the existing model and engine integration. Convert baidu/Unlimited-OCR and publish it to the Hub, preserving the vendor prompt, grounding annotations, tokenizer behavior, and special tokens. Verify that skip_special_tokens=False keeps the grounding markers and that the MLX model produces non-empty OCR output.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.