docling-project / docling-project/docling-parse

Curly double quotes (U+201C/U+201D) are flattened to a single apostrophe, turning inches into feet

Open Beginner friendly
#335 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
333
Forks
80
Avg merge
1d 14h
Merged PRs (30d)
10

Description

The cell text sanitizer replaces the entire curly-quote family with `'`. For the *single* quote family that is reasonable ASCII folding, but the *double* quotes `“ ” „ ‟` also become `'` instead of `"`. That changes meaning: `30.88” x 23.25” x 24”` (inches) is extracted as `30.88' x 23.25' x 24'` (feet). Nothing downstream can recover this — docling, chunkers, and any LLM reading the text see plausible but wrong dimensions.

**Root cause** — `src/parse/pdf_sanitators/constants.h`, `text_constants::replacements`:

```cpp
{"‘", "'"}, // ‘ → ' ok
{"’", "'"}, // ’ → ' ok
{"‛", "'"}, // ‛ → ' ok
{"“", "'"}, // “ → ' should be "
{"”", "'"}, // ” → ' should be "
{"„", "'"}, // „ → ' should be "
{"‟", "'"}, // ‟ → ' should be "
```

Two things make this hard to notice from the outside:

- `sanitize_text(page_cells)` runs unconditionally (`src/parse/pdf_decoders/page.h`), outside the `do_sanitization` flag, so no configuration exposes the pre-sanitized text.
- The exported `orig` field is a copy of the already-sanitized `text` (`src/parse/page_item_sanitators/cells.h`: `item["text"] = cell.text; item["orig"] = cell.text;`), so `orig` misleadingly suggests the parser decoded `'` from the font. The font decode is actually correct — ToUnicode/WinAnsi both produce U+201D.

**Repro** — self-contained: builds a 628-byte PDF using core-14 Helvetica with `/Encoding /WinAnsiEncoding` and no embedded font program, so font decoding cannot be the culprit. The text bytes are `0x94` (quotedblright `”`), `0x92` (quoteright `’`), and a literal `0x27` (`'`).

```python
from pathlib import Path

pdf_path = Path("winansi-quotes.pdf")

# (24\224 x 36\224, 10' poles, it\222s) -> 24" x 36", 10' poles, it's
content = b"BT /F1 18 Tf 72 700 Td (24\224 x 36\224, 10' poles, it\222s) Tj ET"

objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica "
b"/Encoding /WinAnsiEncoding >>",
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"\nendstream",
]

out = bytearray(b"%PDF-1.4\n")
offsets = []
for i, body in enumerate(objects, start=1):
offsets.append(len(out))
out += str(i).encode() + b" 0 obj\n" + body + b"\nendobj\n"

xref_pos = len(out)
out += b"xref\n0 " + str(len(objects) + 1).encode() + b"\n"
out += b"0000000000 65535 f \n"
for off in offsets:
out += f"{off:010d} 00000 n \n".encode()
out += (
b"trailer\n<< /Size " + str(len(objects) + 1).encode() + b" /Root 1 0 R >>\n"
b"startxref\n" + str(xref_pos).encode() + b"\n%%EOF\n"
)
pdf_path.write_bytes(bytes(out))

from docling_parse.pdf_parser import DoclingPdfParser

parser = DoclingPdfParser()
doc = parser.load(path_or_stream=str(pdf_path))
for page_no, page in doc.iterate_pages():
for cell in page.textline_cells:
print(f"page {page_no}: text={cell.text!r} orig={cell.orig!r}")
```

Output:

```
Expected: text="24” x 36”, 10' poles, it’s" (or with ” folded to ": 24" x 36", 10' poles, it's)
Actual: text="24' x 36', 10' poles, it's" orig="24' x 36', 10' poles, it's"
```

Observed on docling-parse 7.15.0 (wheel). The replacement table is unchanged on current `main` (7.16.0).

Contributor guide

Open the contributing guide

Research direction

Start with text_constants::replacements in src/parse/pdf_sanitators/constants.h, then inspect the unconditional call in src/parse/pdf_decoders/page.h and the orig assignment in src/parse/page_item_sanitators/cells.h. Run the self-contained WinAnsi PDF reproduction and verify that curly double quotes remain double quotes while single quotes remain apostrophes in both text and orig.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend
Issue type
Bug
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
90/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.