Add an optional nanonets/Nanonets-OCR-s backend to Docling’s ingestion pipeline so that any PDF or image passed to Docling is converted directly into richly-tagged Markdown.

Requested feature

Add an optional nanonets/Nanonets-OCR-s backend to Docling’s ingestion pipeline so that any PDF or image passed to Docling is converted directly into richly-tagged Markdown.

Docling already excels at parsing diverse document formats and preparing them for Gen-AI workflows, but it still relies on conventional OCR engines that output plain text with limited structural hints.(github.com, docling-project.github.io, research.ibm.com) Integrating Nanonets-OCR-s would let Docling emit Markdown that preserves:

• 
  

  LaTeX maths  — inline $…$ and block $$…$$ equations are reproduced verbatim, so downstream LLMs can render or reason over formulas without heuristic post-processing.(huggingface.co, nanonets.com, news.ycombinator.com)

  
  


• 
  

  Image semantics  — embedded pictures, charts and logos are replaced by <img> tags containing concise, model-generated alt-descriptions, dramatically improving RAG pipelines that mix text and vision.(huggingface.co, reddit.com)

  
  


• 
  

  Signature, watermark & checkbox tags  — legal or business docs arrive with <signature>, <watermark>, ☐/☑/☒ markers already in place, enabling precise filtering or redaction rules.(huggingface.co, huggingface.co)

  
  


• 
  

  HTML/Markdown tables  — complex multi–row or nested tables are emitted twice (HTML + MD), matching Docling’s existing dual-format philosophy and sparing users from brittle table-reconstruction code.(nanonets.com, huggingface.co)

  
  

Why this matters for Docling users

1. 
   

   Greatly reduced post-processing: Docling’s current post-OCR “structure inference” heuristics (e.g., detecting table borders or maths blocks) can be removed or simplified, cutting pipeline runtime and maintenance.(github.com, medium.com)

   
   


2. 
   

   LLM-ready output out-of-the-box: Rich Markdown with semantic tags feeds directly into retrieval-augmented generation, few-shot tutoring or embeddings without extra parsing layers.(huggingface.co, news.ycombinator.com)

   
   


3. 
   

   Consistent developer experience: Nanonets-OCR-s ships via 🤗 Transformers and vLLM, both technologies Docling already uses, so the integration is mostly wiring model invocation plus a feature flag.(huggingface.co, nanonets.com)

   
   


4. 
   

   Future-proofing: The small model runs comfortably on consumer GPUs; when larger variants appear, Docling can expose a --ocr-model switch and scale up without API changes.(nanonets.com, huggingface.co)

   
   

---

Alternatives

Option | Strengths | Shortcomings vs. Nanonets-OCR-s
-- | -- | --
LayoutLMv3 + custom post-processing(arxiv.org, arxiv.org) | Multimodal pre-training; good form understanding | Outputs token positions, not Markdown; LaTeX & checkbox logic must be handcrafted
Amazon Textract(docs.aws.amazon.com, aws.amazon.com, aws.amazon.com) | Reliable table & form extraction; managed scaling | Closed source, pay-per-page, no LaTeX, limited watermark/signature tagging
Google Document AI(cloud.google.com, cloud.google.com, cloud.google.com) | Wide language support; layout detection | Quotas & hard system limits; no Markdown output; LaTeX not preserved
Tesseract + heuristics (status-quo) | Zero cost; already in Docling | Plain text only; fragile table detection; no semantic tags

All alternatives either lack semantic Markdown, require substantial bespoke logic (e.g., to detect $…$ math or image captions), or impose commercial costs and vendor lock-in. Nanonets-OCR-s uniquely combines open-source licensing, lightweight resource needs, and out-of-the-box semantic tagging, making it the most practical upgrade path for Docling today.

### Requested feature

**Add an optional `nanonets/Nanonets-OCR-s` backend to Docling’s ingestion pipeline so that any PDF or image passed to Docling is converted directly into richly-tagged Markdown.**
Docling already excels at parsing diverse document formats and preparing them for Gen-AI workflows, but it still relies on conventional OCR engines that output *plain* text with limited structural hints.([[github.com](https://github.com/docling-project/docling?utm_source=chatgpt.com)][1], [[docling-project.github.io](https://docling-project.github.io/docling/?utm_source=chatgpt.com)][2], [[research.ibm.com](https://research.ibm.com/blog/docling-generative-AI?utm_source=chatgpt.com)][3]) Integrating Nanonets-OCR-s would let Docling emit Markdown that preserves:

* **LaTeX maths**  — inline `$…$` and block `$$…$$` equations are reproduced verbatim, so downstream LLMs can render or reason over formulas without heuristic post-processing.([[huggingface.co](https://huggingface.co/nanonets/Nanonets-OCR-s?utm_source=chatgpt.com)][4], [[nanonets.com](https://nanonets.com/research/nanonets-ocr-s/?utm_source=chatgpt.com)][5], [[news.ycombinator.com](https://news.ycombinator.com/item?id=44287043&utm_source=chatgpt.com)][6])
* **Image semantics**  — embedded pictures, charts and logos are replaced by `` tags containing concise, model-generated alt-descriptions, dramatically improving RAG pipelines that mix text and vision.([[huggingface.co](https://huggingface.co/spaces/Souvik3333/Nanonets-ocr-s?utm_source=chatgpt.com)][7], [[reddit.com](https://www.reddit.com/r/LocalLLaMA/comments/1l9p54x/nanonetsocrs_an_opensource_imagetomarkdown_model/?utm_source=chatgpt.com)][8])
* **Signature, watermark & checkbox tags**  — legal or business docs arrive with ``, ``, `☐/☑/☒` markers already in place, enabling precise filtering or redaction rules.([[huggingface.co](https://huggingface.co/nanonets/Nanonets-OCR-s?utm_source=chatgpt.com)][4], [[huggingface.co](https://huggingface.co/nanonets/Nanonets-OCR-s/resolve/main/README.md?download=true&utm_source=chatgpt.com)][9])
* **HTML/Markdown tables**  — complex multi–row or nested tables are emitted twice (HTML + MD), matching Docling’s existing dual-format philosophy and sparing users from brittle table-reconstruction code.([[nanonets.com](https://nanonets.com/research/nanonets-ocr-s/?utm_source=chatgpt.com)][5], [[huggingface.co](https://huggingface.co/Mungert/Nanonets-OCR-s-GGUF?utm_source=chatgpt.com)][10])

**Why this matters for Docling users**

1. **Greatly reduced post-processing:** Docling’s current post-OCR “structure inference” heuristics (e.g., detecting table borders or maths blocks) can be removed or simplified, cutting pipeline runtime and maintenance.([[github.com](https://github.com/docling-project/docling?utm_source=chatgpt.com)][1], [[medium.com](https://medium.com/%40felix-pappe/pdf-to-markdown-simplified-implementation-and-comparison-of-mistral-and-docling-5c70b6f9a8f0?utm_source=chatgpt.com)][11])
2. **LLM-ready output out-of-the-box:** Rich Markdown with semantic tags feeds directly into retrieval-augmented generation, few-shot tutoring or embeddings without extra parsing layers.([[huggingface.co](https://huggingface.co/nanonets/Nanonets-OCR-s?utm_source=chatgpt.com)][4], [[news.ycombinator.com](https://news.ycombinator.com/item?id=44287043&utm_source=chatgpt.com)][6])
3. **Consistent developer experience:** Nanonets-OCR-s ships via 🤗 Transformers *and* vLLM, both technologies Docling already uses, so the integration is mostly wiring model invocation plus a feature flag.([[huggingface.co](https://huggingface.co/nanonets/Nanonets-OCR-s?utm_source=chatgpt.com)][4], [[nanonets.com](https://nanonets.com/research/nanonets-ocr-s/?utm_source=chatgpt.com)][5])
4. **Future-proofing:** The small model runs comfortably on consumer GPUs; when larger variants appear, Docling can expose a `--ocr-model` switch and scale up without API changes.([[nanonets.com](https://nanonets.com/research/nanonets-ocr-s/?utm_source=chatgpt.com)][5], [[huggingface.co](https://huggingface.co/Mungert/Nanonets-OCR-s-GGUF?utm_source=chatgpt.com)][10])

---

### Alternatives

| Option | Strengths | Shortcomings vs. Nanonets-OCR-s |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------- |
| **LayoutLMv3 + custom post-processing**([[arxiv.org](https://arxiv.org/pdf/2204.08387?utm_source=chatgpt.com)][12], [[arxiv.org](https://arxiv.org/html/2404.10848v1?utm_source=chatgpt.com)][13]) | Multimodal pre-training; good form understanding | Outputs token positions, not Markdown; LaTeX & checkbox logic must be handcrafted |
| **Amazon Textract**([[docs.aws.amazon.com](https://docs.aws.amazon.com/textract/latest/dg/how-it-works-tables.html?utm_source=chatgpt.com)][14], [[aws.amazon.com](https://aws.amazon.com/blogs/machine-learning/announcing-enhanced-table-extractions-with-amazon-textract/?utm_source=chatgpt.com)][15], [[aws.amazon.com](https://aws.amazon.com/textract/features/?utm_source=chatgpt.com)][16]) | Reliable table & form extraction; managed scaling | Closed source, pay-per-page, no LaTeX, limited watermark/signature tagging |
| **Google Document AI**([[cloud.google.com](https://cloud.google.com/document-ai/limits?utm_source=chatgpt.com)][17], [[cloud.google.com](https://cloud.google.com/document-ai/docs/enterprise-document-ocr?utm_source=chatgpt.com)][18], [[cloud.google.com](https://cloud.google.com/document-ai/quotas?utm_source=chatgpt.com)][19]) | Wide language support; layout detection | Quotas & hard system limits; no Markdown output; LaTeX not preserved |
| **Tesseract + heuristics (status-quo)** | Zero cost; already in Docling | Plain text only; fragile table detection; no semantic tags |

All alternatives either lack semantic Markdown, require substantial bespoke logic (e.g., to detect `$…$` math or image captions), or impose commercial costs and vendor lock-in. Nanonets-OCR-s uniquely combines **open-source licensing, lightweight resource needs, and out-of-the-box semantic tagging**, making it the most practical upgrade path for Docling today.

[1]: https://github.com/docling-project/docling?utm_source=chatgpt.com "docling-project/docling: Get your documents ready for gen AI - GitHub"
[2]: https://docling-project.github.io/docling/?utm_source=chatgpt.com "Docling - GitHub Pages"
[3]: https://research.ibm.com/blog/docling-generative-AI?utm_source=chatgpt.com "IBM is open-sourcing a new toolkit for document conversion"
[4]: https://huggingface.co/nanonets/Nanonets-OCR-s?utm_source=chatgpt.com "nanonets/Nanonets-OCR-s - Hugging Face"
[5]: https://nanonets.com/research/nanonets-ocr-s/?utm_source=chatgpt.com "Advanced Document Understanding API - Nanonets OCR"
[6]: https://news.ycombinator.com/item?id=44287043&utm_source=chatgpt.com "OCR model that transforms documents into structured markdown"
[7]: https://huggingface.co/spaces/Souvik3333/Nanonets-ocr-s?utm_source=chatgpt.com "Nanonets Ocr S - a Hugging Face Space by Souvik3333"
[8]: https://www.reddit.com/r/LocalLLaMA/comments/1l9p54x/nanonetsocrs_an_opensource_imagetomarkdown_model/?utm_source=chatgpt.com "Nanonets-OCR-s: An Open-Source Image-to-Markdown Model with ..."
[9]: https://huggingface.co/nanonets/Nanonets-OCR-s/resolve/main/README.md?download=true&utm_source=chatgpt.com "6.29 kB - Hugging Face"
[10]: https://huggingface.co/Mungert/Nanonets-OCR-s-GGUF?utm_source=chatgpt.com "Mungert/Nanonets-OCR-s-GGUF - Hugging Face"
[11]: https://medium.com/%40felix-pappe/pdf-to-markdown-simplified-implementation-and-comparison-of-mistral-and-docling-5c70b6f9a8f0?utm_source=chatgpt.com "PDF to Markdown: Mistral vs. Docling OCR | Medium"
[12]: https://arxiv.org/pdf/2204.08387?utm_source=chatgpt.com "[PDF] LayoutLMv3: Pre-training for Document AI with Unified Text ... - arXiv"
[13]: https://arxiv.org/html/2404.10848v1?utm_source=chatgpt.com "A LayoutLMv3-Based Model for Enhanced Relation Extraction in ..."
[14]: https://docs.aws.amazon.com/textract/latest/dg/how-it-works-tables.html?utm_source=chatgpt.com "Tables - Amazon Textract - AWS Documentation"
[15]: https://aws.amazon.com/blogs/machine-learning/announcing-enhanced-table-extractions-with-amazon-textract/?utm_source=chatgpt.com "Announcing enhanced table extractions with Amazon Textract - AWS"
[16]: https://aws.amazon.com/textract/features/?utm_source=chatgpt.com "Amazon Textract features"
[17]: https://cloud.google.com/document-ai/limits?utm_source=chatgpt.com "Limits | Document AI - Google Cloud"
[18]: https://cloud.google.com/document-ai/docs/enterprise-document-ocr?utm_source=chatgpt.com "Enterprise Document OCR | Document AI | Google Cloud"
[19]: https://cloud.google.com/document-ai/quotas?utm_source=chatgpt.com "Quotas | Document AI - Google Cloud"