docling-project / docling-project/docling

Feature Request: Auto-detect corrupted PDF text layers and fallback to OCR

Open
#2,963 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

## Problem Description

When processing PDFs that have **CID-keyed fonts without ToUnicode mappings**, Docling extracts garbage text instead of readable content.

### Example Output (Before Fix)
```
/31 /8 /18 /12 /18 /5 /8 /12 /28 /i255 /32 /8 /21 /5 /28 /22 /21 /6 /13 /4 /i255 /25 /15 /17 /31 /32...
```

This occurs because:
1. The PDF uses CID (Character ID) fonts for glyph storage
2. The `/ToUnicode` CMap is missing from the PDF
3. Docling's text extraction pulls the raw CID codes instead of actual characters

### Current Workaround

The issue can be resolved by forcing full-page OCR:

```python
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions, TesseractCliOcrOptions

pipeline_options = PdfPipelineOptions()
pipeline_options.do_ocr = True
pipeline_options.do_table_structure = False # Table structure may pull bad text
pipeline_options.images_scale = 2.0
pipeline_options.generate_page_images = True
pipeline_options.ocr_options = TesseractCliOcrOptions()
pipeline_options.ocr_options.force_full_page_ocr = True

converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
}
)
```

### Feature Request

1. **Auto-detection**: Implement a heuristic to detect corrupted text layers. For example:
- If >50% of extracted characters are CID escape codes (e.g., `/\d+` or `/i\d+`)
- If extracted text has unusually low printable ASCII ratio
- Automatically fall back to `force_full_page_ocr=True`

2. **Documentation**: Add a troubleshooting section for "garbage output" scenarios

3. **CLI Flag**: Expose `--force-ocr` as a command-line option for the Docling CLI

### Environment
- **Docling Version**: Latest (installed via `uv run --with docling`)
- **OS**: macOS
- **OCR Engine**: TesseractCliOcrOptions

### Affected PDFs
This issue is common in:
- Government/institutional PDFs
- PDFs created by older PDF generators
- PDFs with embedded subset fonts

---

Thank you for the excellent library! This feature would make Docling more robust for real-world PDF processing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.