docling-project / docling-project/docling
Feature Request: Auto-detect corrupted PDF text layers and fallback to OCR
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
## Problem Description
When processing PDFs that have **CID-keyed fonts without ToUnicode mappings**, Docling extracts garbage text instead of readable content.
### Example Output (Before Fix)
```
/31 /8 /18 /12 /18 /5 /8 /12 /28 /i255 /32 /8 /21 /5 /28 /22 /21 /6 /13 /4 /i255 /25 /15 /17 /31 /32...
```
This occurs because:
1. The PDF uses CID (Character ID) fonts for glyph storage
2. The `/ToUnicode` CMap is missing from the PDF
3. Docling's text extraction pulls the raw CID codes instead of actual characters
### Current Workaround
The issue can be resolved by forcing full-page OCR:
```python
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions, TesseractCliOcrOptions
pipeline_options = PdfPipelineOptions()
pipeline_options.do_ocr = True
pipeline_options.do_table_structure = False # Table structure may pull bad text
pipeline_options.images_scale = 2.0
pipeline_options.generate_page_images = True
pipeline_options.ocr_options = TesseractCliOcrOptions()
pipeline_options.ocr_options.force_full_page_ocr = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
}
)
```
### Feature Request
1. **Auto-detection**: Implement a heuristic to detect corrupted text layers. For example:
- If >50% of extracted characters are CID escape codes (e.g., `/\d+` or `/i\d+`)
- If extracted text has unusually low printable ASCII ratio
- Automatically fall back to `force_full_page_ocr=True`
2. **Documentation**: Add a troubleshooting section for "garbage output" scenarios
3. **CLI Flag**: Expose `--force-ocr` as a command-line option for the Docling CLI
### Environment
- **Docling Version**: Latest (installed via `uv run --with docling`)
- **OS**: macOS
- **OCR Engine**: TesseractCliOcrOptions
### Affected PDFs
This issue is common in:
- Government/institutional PDFs
- PDFs created by older PDF generators
- PDFs with embedded subset fonts
---
Thank you for the excellent library! This feature would make Docling more robust for real-world PDF processing.
Contributor guide
Assessment
This issue has not been assessed yet.