docling-project / docling-project/docling
Full-page OCR is blocked by UnicodeDecodeError in native PDF text parsing
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Bug
A PDF page renders successfully as an image, but native text parsing raises a
`UnicodeDecodeError` before full-page OCR can run.
The conversion is configured with `OcrMode.FULL_PAGE`. Since the page image is
valid and full-page OCR does not rely on native PDF text for its final text
cells, I expected OCR to proceed or the native text layer to degrade
gracefully.
Instead, the entire page conversion fails during page preprocessing.
This may be related to #1111, but this case uses the current
`docling_parse.pdf_parser._to_cells_from_decoder()` path and specifically
prevents full-page OCR from running.
### Steps to reproduce
```python
from pathlib import Path
from docling.backend.docling_parse_backend import DoclingParseDocumentBackend
from docling.datamodel.accelerator_options import (
AcceleratorDevice,
AcceleratorOptions,
)
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import (
OcrMode,
PdfPipelineOptions,
RapidOcrOptions,
)
from docling.document_converter import DocumentConverter, PdfFormatOption
source = Path("202400733.pdf")
options = PdfPipelineOptions()
options.do_ocr = True
options.do_table_structure = False # isolate the preprocessing/OCR failure
options.ocr_options = RapidOcrOptions(
mode=OcrMode.FULL_PAGE,
backend="torch",
)
options.accelerator_options = AcceleratorOptions(
device=AcceleratorDevice.CPU,
)
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(
pipeline_options=options,
backend=DoclingParseDocumentBackend,
)
}
)
result = converter.convert(source, page_range=(1, 1))
print(result.document.export_to_markdown())
The same underlying failure can be reproduced directly with docling-parse:
from pathlib import Path
from docling_parse.pdf_parser import DoclingPdfParser
source = Path("202400733.pdf")
document = DoclingPdfParser().load(source)
document.get_page(1)
```
### Actual behavior
Page 1 fails before OCR with:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf
in position 0: invalid continuation byte
Relevant stack:
docling/models/stages/page_preprocessing/page_preprocessing_model.py
page._backend.get_segmented_page()
docling/backend/docling_parse_backend.py
self._ensure_parsed()
self._dp_doc.get_page(...)
docling_parse/pdf_parser.py
segmented_page_from_decoder(...)
_to_cells_from_decoder(...)
text=cell.text
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf
in position 0: invalid continuation byte
The page image itself can be rendered successfully. Running OCR after replacing
only the failed native text layer with an empty text layer produces OCR text,
so the rasterized page is not damaged.
### Expected behavior
When full-page OCR is explicitly requested and page rendering succeeds, a
native PDF text decoding failure should not prevent OCR from running.
Possible behavior could be:
1. Attempt native text parsing normally.
2. If native text decoding fails in OcrMode.FULL_PAGE, emit a warning.
3. Continue with an empty native text layer and run full-page OCR.
4. Preserve the existing exception behavior for other OCR modes.
I am not sure whether invalid text cells should instead be handled inside
docling-parse, so guidance on the appropriate ownership would be appreciated.
### Additional observation
After locally bypassing the preprocessing failure, table structure processing
can call page._backend.get_segmented_page() again and trigger the same
decoding error.
It may be necessary for downstream stages to use the OCR-updated
page.parsed_page rather than reparsing the native backend page.
### Environment
Reproduced with:
Docling: 2.115.0
Docling Core: 2.87.1
Docling Parse: 7.8.1
Python: 3.11
Platform: Linux container
Also reproduced directly with docling-parse using:
Docling main base commit: 873f9902
Docling Slim: 2.115.0
Docling Core: 2.86.0
Docling Parse: 7.7.0
Python: 3.13.2
Platform: Windows
### Sample file
[202400733.pdf](https://github.com/user-attachments/files/30398850/202400733.pdf)
Failure occurs on page 1 of 202400733.pdf.
SHA-256:
5DA5D671FFD685F835D67FE0C39C0D8D6BC36F94C79CAC83C11796242CEEE896
The original PDF is attached without rewriting or extracting pages, since
rewriting the PDF may normalize the malformed text data and hide the issue.
Contributor guide
Research direction
Start with docling/models/stages/page_preprocessing/page_preprocessing_model.py and docling/backend/docling_parse_backend.py, then trace docling_parse/pdf_parser.py through _to_cells_from_decoder(). Reproduce with page 1 of 202400733.pdf using the supplied converter and direct DoclingPdfParser examples. Done means full-page OCR proceeds after a native decoding failure while other OCR modes retain their existing behavior, including later page access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100