docling-project / docling-project/docling

Inconsistent markdown in linux env vs windows env

Open
#1,259 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug
When converting a pdf document to markdown locally, I am seeing inconsistent outputs in a Linux environment versus Windows. In a Windows environment, the markdown seems to be correctly formatted. Take a look at the attached markdown for section 7 and 8. In the windows md file these sections gets extracted cleanly. However, in the linux markdown section 7 and 8 get stuck in a table that shouldn't be there. It's important that the linux markdown is consistent as our production container apps hosting docling are linux-based.
### Steps to reproduce
Code snippet used to generate markdown along with the pdf file and output md files:

[wa2.pdf](https://github.com/user-attachments/files/19508960/wa2.pdf)

[windows_docling_pdf_to_markdown.md](https://github.com/user-attachments/files/19508967/windows_docling_pdf_to_markdown.md)

[linux_docling_pdf_to_markdown.md](https://github.com/user-attachments/files/19508987/linux_docling_pdf_to_markdown.md)

```
import os
import asyncio
import html
from docling.document_converter import DocumentConverter, PdfFormatOption, WordFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import (
PdfPipelineOptions,
PipelineOptions,
RapidOcrOptions,
AcceleratorOptions,
AcceleratorDevice
)
from huggingface_hub import snapshot_download
import onnxruntime
import torch

async def convert_document_to_markdown(source: str) -> str:
"""
Convert a document (PDF or DOCX) to markdown text using Docling.
If the source is a PDF and the initial conversion produces empty markdown,
reprocess using RapidOCR.

Args:
source (str): The path to the document file.

Returns:
str: The converted markdown text.
"""
ext = os.path.splitext(source)[1].lower()

# Set up common accelerator options to use GPU
accelerator_options = AcceleratorOptions(
num_threads=6,
device=AcceleratorDevice.CUDA # Use CUDA-enabled GPU
)

# Initialize the DocumentConverter with pipeline options based on file type.
if ext == ".pdf":
# PDF conversion without OCR (initial attempt)
pipeline_options = PdfPipelineOptions()
pipeline_options.accelerator_options = accelerator_options
pipeline_options.do_ocr = False # Do not run OCR on first pass
pipeline_options.do_table_structure = True
pipeline_options.table_structure_options.do_cell_matching = True

doc_converter = DocumentConverter(
format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)}
)
elif ext == ".docx":
pipeline_options = PipelineOptions() # DOCX-specific pipeline options if desired.
pipeline_options.accelerator_options = accelerator_options
doc_converter = DocumentConverter(
format_options={InputFormat.DOCX: WordFormatOption(pipeline_options=pipeline_options)}
)
else:
raise ValueError(f"Unsupported file type: {ext}")

# Process the document using the initial settings.
doc = await asyncio.to_thread(doc_converter.convert, source)
markdown_text = doc.document.export_to_markdown()
print("Initial Markdown output:")
print(markdown_text)

# If the file is a PDF and the extracted Markdown is empty, reprocess with RapidOCR.
if ext == ".pdf" and not markdown_text.strip():
print("Empty markdown detected. Reprocessing using RapidOCR...")

# Download RapidOCR models from HuggingFace (if not already cached)
download_path = await asyncio.to_thread(snapshot_download, repo_id="SWHL/RapidOCR")
det_model_path = os.path.join(download_path, "PP-OCRv4", "en_PP-OCRv3_det_infer.onnx")
rec_model_path = os.path.join(download_path, "PP-OCRv3", "en_PP-OCRv3_rec_infer.onnx")
cls_model_path = os.path.join(download_path, "PP-OCRv3", "ch_ppocr_mobile_v2.0_cls_train.onnx")

pipeline_options = PdfPipelineOptions()
pipeline_options.accelerator_options = accelerator_options
pipeline_options.do_ocr = True # Enable OCR this time
pipeline_options.do_table_structure = True
pipeline_options.table_structure_options.do_cell_matching = True
pipeline_options.ocr_options = RapidOcrOptions(
force_full_page_ocr=True,
det_model_path=det_model_path,
rec_model_path=rec_model_path,
cls_model_path=cls_model_path
)

doc_converter = DocumentConverter(
format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)}
)

doc = await asyncio.to_thread(doc_converter.convert, source)
markdown_text = doc.document.export_to_markdown()
print("Markdown output after RapidOCR:")
print(markdown_text)

return markdown_text

markdown_text = await convert_document_to_markdown("wa2.pdf")
```

### Docling version
For windows env:
Docling version: 2.28.2
Docling Core version: 2.23.3
Docling IBM Models version: 3.4.1
Docling Parse version: 4.0.0
Python: cpython-312 (3.12.6)
Platform: Windows-11-10.0.22631-SP0

For linux env:
Docling version: 2.28.2
Docling Core version: 2.24.0
Docling IBM Models version: 3.4.1
Docling Parse version: 4.0.0
Python: cpython-312 (3.12.9)
Platform: Linux-5.15.173.1-1.cm2-x86_64-with-glibc2.36
...

### Python version
python-3.12
...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.