docling-project / docling-project/docling

PDF: URL char-by-char spacing + metadata field leakage in references

Open
#3,882 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug

Two related text-spacing artifacts in PDF extraction:

**Artifact 1 — URL character spacing:** URLs in reference sections are rendered with single-character spacing, where each character is separated by a space:
```
h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / j . p e q u i n . 2 0 2 3 . 1 2 . 0 0 2
```

**Artifact 2 — Metadata field leakage:** PDF metadata field names (WGROUP, STRING, PUBLICATION, PAGE, etc.) are extracted as spaced-character text and appear in the references section:
```
W G R O U P : S T R I N G : P U B L I C A T I O N
P A G E : S T R I N G : A R T I C L E / C H A P T E R
```

### Steps to reproduce

```python
from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("verkerk_forensic_necropsies_2026.pdf")
md = result.document.export_to_markdown()

# Artifact 1: URLs have character spacing
assert "h t t p s" in md # Bug: URL mangled

# Artifact 2: metadata fields leaked as text
assert "W G R O U P" in md # Bug: metadata leaked
```

### Expected behavior

1. URLs should be extracted as clean URLs: `https://doi.org/10.1016/...`
2. PDF metadata fields should not appear in the text output — they are structural, not content

### Source location

- **URL spacing**: `page_assemble_model.py:118 sanitize_text` — no URL-aware run un-spacing. The PDF producer stored URLs with explicit character spacing (one glyph per `Tj` operator), and the extractor normalized these to word breaks.
- **Metadata leakage**: `backend/docling_parse_backend.py` — the backend pulls all text from the page without discarding `/Artifact` marked-content or non-`Contents` `Tj` ops.

### Suggestion

1. For URLs: detect runs matching URL patterns (`https?://`, `doi.org`) and collapse inter-character spaces
2. For metadata: discard text runs matching known PDF metadata field patterns (single-char-spaced uppercase sequences like `Type`, `Subtype`, `Filter`, `WGROUP`, `STRING`)

### Docling version

2.115.0

### Regression test PDF

- `verkerk_forensic_necropsies_2026.pdf` (4.4 MB) — Springer journal article, both artifacts visible in the references section

Contributor guide

Open the contributing guide

Research direction

Read page_assemble_model.py around sanitize_text and backend/docling_parse_backend.py, then run the supplied DocumentConverter reproduction with verkerk_forensic_necropsies_2026.pdf. Done means URLs export without character spacing and PDF metadata field names no longer appear in the extracted references text.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.