docling-project / docling-project/docling
Unlimited-OCR: MLX conversion for Apple silicon
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
Split out of #3679, where @geoHeil asked whether Unlimited-OCR could be converted to MLX and published
to the Hub so that docling keeps its "most models also run on Apple silicon" property.
Keeping it separate so the parsing work in #3679 stays reviewable on its own — that change is
API-first and does not depend on this one.
### What would be needed
- an MLX conversion of [`baidu/Unlimited-OCR`](https://huggingface.co/baidu/Unlimited-OCR) (MIT
licensed, so redistribution of a converted copy is permitted) published to the Hub;
- an `engine_overrides` entry for `VlmEngineType.MLX` on the `unlimited_ocr` preset pointing at that
repo id;
- a check that the model still emits its grounding annotations under MLX. This matters more than usual
here: the model has no chat template and is very sensitive to prompt wording — the vendor prompt
`\n<|grounding|>OCR this image.` works, while generic "convert to markdown" phrasings return an
empty completion with `finish_reason: stop` and no error. A conversion that silently changes tokenizer
behaviour would look like an empty page rather than a failure.
- the same for `skip_special_tokens`: the grounding markers are special tokens, and on the
OpenAI-compatible path they are stripped unless `skip_special_tokens=False` is passed.
### Status
We have Apple silicon available and intend to look at this, but not as part of the parsing PR. Anyone
who gets there first is welcome — please comment here so the work is not duplicated.
Contributor guide
Research direction
Start with the unlimited_ocr preset and its engine_overrides entry for VlmEngineType.MLX, then inspect the existing model and engine integration. Convert baidu/Unlimited-OCR and publish it to the Hub, preserving the vendor prompt, grounding annotations, tokenizer behavior, and special tokens. Verify that skip_special_tokens=False keeps the grounding markers and that the MLX model produces non-empty OCR output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100