docling-project / docling-project/docling-parse

Merged textline cells decode every run with one font, corrupting Symbol glyphs (→ becomes "fi")

Open
#317 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
333
Forks
80
Avg merge
1d 14h
Merged PRs (30d)
10

Description

### Summary

When adjacent text runs are merged into a single textline cell, every byte in that cell is decoded using one font — the one the cell reports as `font_key` — rather than the font each run was actually drawn in. A `/Symbol` run sharing a line with Helvetica text is therefore read through `Helvetica.afm`, where code 174 is `fi` instead of `arrowright`.

A source `Alpha → Bravo` comes out as `Alpha fi Bravo`. This is a quiet failure: the arrow becomes a plausible-looking *word*, so it survives review and silently degrades any downstream embedding or quoted citation, rather than surfacing as an obvious `GLYPH<…>` placeholder.

Runs that are **not** merged decode correctly, so this is not a missing Symbol encoding — `Symbol.afm` ships and is used correctly in the unmerged case.

### Versions

```
docling-parse == 7.8.0
docling == 2.113.0 (default DoclingParseV4 backend)
pypdf == 6.14.2
pypdfium2 == 5.12.0
Python 3.14.3, Windows
```

### Reproduction

```python
from docling_parse.pdf_parser import DoclingPdfParser

doc = DoclingPdfParser().load(path_or_stream="symbol-encoding-repro.pdf")
for _n, page in doc.iterate_pages():
for cell in page.textline_cells:
if "fi" in cell.text:
print(repr(cell.font_key), repr(cell.text))
```

**Actual**

```
'/F1' 'HotelHote fi Ind'
'/F1' 'Kilo fi Lim'
'/F1' 'NovemberNove fi Osc'
```

**Expected** — `fi` should be `→` (U+2192) in each.

`pypdf`, `pypdfium2` and docling's own `PyPdfiumDocumentBackend` all decode the same bytes correctly.

### What the PDF contains

A ReportLab-produced page that switches font mid-line to draw the arrow:

```
BT 1 0 0 1 176.0787 96.5 Tm (HotelHote ) Tj /F5 8.5 Tf 12 TL (®) Tj /F1 8.5 Tf 12 TL ( Ind) Tj T* ET
```

`/F5` is `/Symbol` with no `/Encoding` and no `/ToUnicode` — spec-legal, and it obliges the reader to use the font's built-in encoding. `/F1` is `/Helvetica` with `/Encoding /WinAnsiEncoding`.

### Analysis

The merged cell reports `font_key='/F1'`, and the byte drawn in `/F5` is resolved through `/F1`'s metrics. Comparing the shipped AFMs explains both glyphs that come out wrong:

| AFM | code 174 (0xAE) | code 72 (0x48) |
|---|---|---|
| `fonts/standard/Helvetica.afm` | `fi` (line 134) | `H` (line 66) |
| `fonts/standard/Symbol.afm` | `arrowright` (line 131) | — |
| `fonts/standard/ZapfDingbats.afm` | — | `a35`, a star (line 62) |

- The `/Symbol` arrow (174) read through Helvetica → `fi` ✔ matches observed output
- A `/ZapfDingbats` star (72) read through Helvetica → `H` ✔ same document, same cause

Note the AFM also wins over the `/WinAnsiEncoding` that `/F1` explicitly declares — under WinAnsi, 174 would be `®`, not `fi`. That places this in the same wrong-table-wins family as #187 (*MacRomanEncoding ignored when font matches known base font*), which points at `get_correct_character()` in `src/v2/pdf_resources/page_font.h` (lines 480–498). This report is a distinct case though: no declared encoding is being ignored here — the problem is that a merged cell applies one run's font to bytes belonging to another.

### It is merging-specific

A one-page PDF with the *same* font setup and the same `(®) Tj` in `/F5`, but whose runs stay in separate cells, decodes **correctly** to `→`. The encoding machinery is fine in isolation; only the merged path is affected.

### Suggested direction

Carry the per-run font through to character resolution so each byte is decoded with the font it was drawn in, rather than resolving a merged cell against a single `font_key`. Failing that, avoid merging runs across a font change when the fonts resolve characters differently.

symbol-encoding-repro.pdf (base64, 4865 bytes)

```
JVBERi0xLjQKJeLjz9MKMSAwIG9iago8PAovUHJvZHVjZXIgKHB5cGRmKQo+PgplbmRvYmoKMiAwIG9iago8PAovVHlwZSAvUGFnZXMKL0NvdW50IDEKL0tp
ZHMgWyA0IDAgUiBdCj4+CmVuZG9iagozIDAgb2JqCjw8Ci9UeXBlIC9DYXRhbG9nCi9QYWdlcyAyIDAgUgo+PgplbmRvYmoKNCAwIG9iago8PAovQ29udGVu
dHMgNSAwIFIKL01lZGlhQm94IFsgMCAwIDU5NS4yNzU2IDg0MS44ODk4IF0KL1Jlc291cmNlcyA8PAovRm9udCA2IDAgUgovUHJvY1NldCBbIC9QREYgL1Rl
eHQgL0ltYWdlQiAvSW1hZ2VDIC9JbWFnZUkgXQo+PgovUm90YXRlIDAKL1RyYW5zIDw8Cj4+Ci9UeXBlIC9QYWdlCi9QYXJlbnQgMiAwIFIKPj4KZW5kb2Jq
CjUgMCBvYmoKPDwKL0xlbmd0aCAzNTY1Cj4+CnN0cmVhbQoxIDAgMCAxIDAgMCBjbSAgQlQgL0YxIDEyIFRmIDE0LjQgVEwgRVQKcQpCVCAvRjEgNy41IFRm
IDkgVEwgRVQKLjQ2NjY2NyAuNDY2NjY3IC40NjY2NjcgcmcKQlQgMSAwIDAgMSA0NS4zNTQzMyAyNS41MTE4MSBUbSAoQWxwaGFBbHBoYUFscGhhQWxwaGFB
bHBoYUFscGhhQWxwaGFBbHBoYUFscGhhQWxwaGFBbHBoYUFscGgpIFRqIFQqIEVUCkJUIDEgMCAwIDEgNTI2LjE1MzggMjUuNTExODEgVG0gKEJyYXZvQikg
VGogVCogRVQKUQpxCjEgMCAwIDEgNTEuMzU0MzMgNzYyLjUzNTQgY20KcQowIDAgMCByZwpCVCAxIDAgMCAxIDAgMTguNSBUbSAvRjEgOS41IFRmIDE0IFRM
IChDaGFybGllQ2hhcmxpZUNoYXJsaWVDaGFybGllQ2hhcmxpZUNoYXJsaWVDaGFybGllQ2hhcmxpZUNoYXJsaWVDaGFybGllQ2hhcmxpZUNoYXJsaWVDaGFy
bGllQ2hhcmxpZUNoYXJsaWVDaGFyKSBUaiBUKiAoRGVsdGFEZWx0YURlbHRhRGVsdGFEZWx0KSBUaiBUKiBFVApRClEKcQoxIDAgMCAxIDcwLjg2NjE0IDYz
MC41MzU0IGNtCnEKLjkxNzY0NyAuOTE3NjQ3IC45MTc2NDcgcmcKbiAwIDEyNiA0NTMuNTQzMyAtMTggcmUgZioKMSAxIDEgcmcKbiAwIDEwOCA0NTMuNTQz
MyAtMTggcmUgZioKLjk3MjU0OSAuOTcyNTQ5IC45NzI1NDkgcmcKbiAwIDkwIDQ1My41NDMzIC0xOCByZSBmKgoxIDEgMSByZwpuIDAgNzIgNDUzLjU0MzMg
LTE4IHJlIGYqCi45NzI1NDkgLjk3MjU0OSAuOTcyNTQ5IHJnCm4gMCA1NCA0NTMuNTQzMyAtMTggcmUgZioKMSAxIDEgcmcKbiAwIDM2IDQ1My41NDMzIC0x
OCByZSBmKgouOTcyNTQ5IC45NzI1NDkgLjk3MjU0OSByZwpuIDAgMTggNDUzLjU0MzMgLTE4IHJlIGYqCjAgMCAwIHJnCkJUIC9GMiA4LjUgVGYgMTIgVEwg
RVQKQlQgMSAwIDAgMSA2IDExNC41IFRtIChFY2hvRWNoKSBUaiBUKiBFVApCVCAxIDAgMCAxIDE3Ni4wNzg3IDExNC41IFRtIChGb3h0cm90Rm94dHJvdCkg
VGogVCogRVQKQlQgL0YxIDguNSBUZiAxMiBUTCBFVApCVCAxIDAgMCAxIDYgOTYuNSBUbSAoR29sZkdvbCkgVGogVCogRVQKQlQgMSAwIDAgMSAxNzYuMDc4
NyA5Ni41IFRtIChIb3RlbEhvdGUgKSBUaiAvRjUgOC41IFRmIDEyIFRMIChcMjU2KSBUaiAvRjEgOC41IFRmIDEyIFRMICggSW5kKSBUaiBUKiBFVApCVCAx
IDAgMCAxIDYgNzguNSBUbSAoSnVsaWV0SnUpIFRqIFQqIEVUCkJUIDEgMCAwIDEgMTc2LjA3ODcgNzguNSBUbSAoS2lsbyApIFRqIC9GNSA4LjUgVGYgMTIg
VEwgKFwyNTYpIFRqIC9GMSA4LjUgVGYgMTIgVEwgKCBMaW0pIFRqIFQqIEVUCkJUIDEgMCAwIDEgNiA2MC41IFRtIChNaWtlTWlrZU1pa2VNaWtlTWkpIFRq
IFQqIEVUCkJUIDEgMCAwIDEgMTc2LjA3ODcgNjAuNSBUbSAoTm92ZW1iZXJOb3ZlICkgVGogL0Y1IDguNSBUZiAxMiBUTCAoXDI1NikgVGogL0YxIDguNSBU
ZiAxMiBUTCAoIE9zYykgVGogVCogRVQKQlQgMSAwIDAgMSA2IDQyLjUgVG0gKFBhcGFQYXBhUGFwYVBhcGFQYSkgVGogVCogRVQKQlQgMSAwIDAgMSAxNzYu
MDc4NyA0Mi41IFRtIChBbHBoYUFscGggKSBUaiAvRjUgOC41IFRmIDEyIFRMIChcMjU2KSBUaiAvRjEgOC41IFRmIDEyIFRMICggQnJhKSBUaiBUKiBFVApC
VCAxIDAgMCAxIDYgMjQuNSBUbSAoQ2hhcmxpZUNoYXJsaWVDaGFybGkpIFRqIFQqIEVUCkJUIDEgMCAwIDEgMTc2LjA3ODcgMjQuNSBUbSAoRGVsdCApIFRq
IC9GNSA4LjUgVGYgMTIgVEwgKFwyNTYpIFRqIC9GMSA4LjUgVGYgMTIgVEwgKCBFY2gpIFRqIFQqIEVUCkJUIDEgMCAwIDEgNiA2LjUgVG0gKEZveHRyb3RG
b3h0cm90RikgVGogVCogRVQKQlQgMSAwIDAgMSAxNzYuMDc4NyA2LjUgVG0gKEdvbGZHb2xmICkgVGogL0Y1IDguNSBUZiAxMiBUTCAoXDI1NikgVGogL0Yx
IDguNSBUZiAxMiBUTCAoIEhvdCkgVGogVCogRVQKcQoxIEoKMSBqCi43MzMzMzMgLjczMzMzMyAuNzMzMzMzIFJHCi4zIHcKbiAwIDEyNiBtIDQ1My41NDMz
IDEyNiBsIFMKbiAwIDAgbSA0NTMuNTQzMyAwIGwgUwpuIDAgMCBtIDAgMTI2IGwgUwpuIDQ1My41NDMzIDAgbSA0NTMuNTQzMyAxMjYgbCBTCm4gMCAxMDgg
bSA0NTMuNTQzMyAxMDggbCBTCm4gMCA5MCBtIDQ1My41NDMzIDkwIGwgUwpuIDAgNzIgbSA0NTMuNTQzMyA3MiBsIFMKbiAwIDU0IG0gNDUzLjU0MzMgNTQg
bCBTCm4gMCAzNiBtIDQ1My41NDMzIDM2IGwgUwpuIDAgMTggbSA0NTMuNTQzMyAxOCBsIFMKbiAxNzAuMDc4NyAwIG0gMTcwLjA3ODcgMTI2IGwgUwpRClEK
UQpxCjEgMCAwIDEgNTEuMzU0MzMgNjIyLjUzNTQgY20KUQpxCjEgMCAwIDEgNTEuMzU0MzMgNjAxLjUzNTQgY20KcQouNjY2NjY3IC42NjY2NjcgLjY2NjY2
NyBSRwouNSB3Cm4gMCAtOCA0OTIuNTY2OSAzMSByZSBTClEKcQpCVCAxIDAgMCAxIDggNSBUbSAxNSBUTCAvRjIgMTAgVGYgMCAwIDAgcmcgKEluZGlhSW5k
aWFJbmRpYUluZGlhSW5kaWFJbmRpYUkpIFRqIC9GMSAxMCBUZiAoIEp1bGlldEp1bGlldEp1bGlldEp1bGlldEp1bGlldEp1bGlldEp1bGlldEp1bGlldEp1
bGlldEp1bGllKSBUaiBUKiBFVApRClEKcQoxIDAgMCAxIDUxLjM1NDMzIDUzMS41MzU0IGNtCnEKLjY2NjY2NyAuNjY2NjY3IC42NjY2NjcgUkcKLjUgdwpu
IDAgLTggNDkyLjU2NjkgNzYgcmUgUwpRCnEKQlQgMSAwIDAgMSA4IDUwIFRtIDE1IFRMIC9GMiAxMCBUZiAwIDAgMCByZyAoS2lsb0tpbG9LaWxvS2lsb0tp
bG9LKSBUaiAvRjEgMTAgVGYgKCBMaW1hTGltYUxpbWFMaW1hTGltYUxpbWFMaW1hTGltYUxpbWFMaW1hTGltYUxpbWFMaW1hTGltYUxpbWFMaW1hTGltYUxp
bWFMaW1hTGltYUxpbWFMKSBUaiBUKiAoTWlrZU1pa2VNaWtlTWlrZU1pa2VNaWtlTWlrZU1pa2VNaWtlTWlrZU1pa2VNaWtlTWlrZU1pa2VNaWtlTWlrZU1p
a2VNaWtlTWlrZU1pa2VNaWtlTWlrZU1pa2UpIFRqIFQqIChOb3ZlbWJlck5vdmVtYmVyTm92ZW1iZXJOb3ZlbWJlck5vdmVtYmVyTm92ZW1iZXJOb3ZlbWJl
ck5vdmVtYmVyTm92ZW1iZXJOb3ZlbWJlck5vdmVtYmVyTm92ZW1iZSkgVGogVCogKE9zY2FyT3NjYXJPc2Nhck9zKSBUaiBUKiBFVApRClEKcQoxIDAgMCAx
IDUxLjM1NDMzIDQ5OC41MzU0IGNtCnEKMCAwIDAgcmcKQlQgMSAwIDAgMSAwIDQgVG0gL0YyIDE3IFRmIDIxIFRMIChQYXBhUGFwYVBhcGFQYXBhUGFwYVBh
cGFQYXBhUGFwYVBhcGFQYXBhKSBUaiBUKiBFVApRClEKcQoxIDAgMCAxIDUxLjM1NDMzIDQ0OC41MzU0IGNtCnEKMCAwIDAgcmcKQlQgMSAwIDAgMSAwIDMy
LjUgVG0gL0YxIDkuNSBUZiAxNCBUTCAoQWxwaGFBbHBoYUFscGhhQWxwaGFBbHBoYUFscGhhQWxwaGFBbHBoYUFscGhhQWxwaGFBbHBoYUFscGhhQWxwaGFB
bHBoYUFscGhhQWxwaGFBbHBoYUFscGhhQWxwaGFBbHBoYUFscGhhQWxwKSBUaiBUKiAoQnJhdm9CcmF2b0JyYXZvQnJhdm9CcmF2b0JyYXZvQnJhdm9CcmF2
b0JyYXZvQnJhdm9CcmF2b0JyYXZvQnJhdm9CcmF2b0JyYXZvQnJhdm9CcmF2b0JyYXZvQnJhdm9CcmF2b0JyYXZvKSBUaiBUKiAoQ2hhcmxpZUNoYXJsaWVD
aGFybGllQ2hhcmxpZUNoYXJsaWVDaGFybGllQ2hhcmxpZUNoYXJsaWVDaGFybGllQ2hhcmxpZUNoYXJsaWVDaGFybGllQ2hhcikgVGogVCogRVQKUQpRCiAK
CmVuZHN0cmVhbQplbmRvYmoKNiAwIG9iago8PAovRjEgNyAwIFIKL0YyIDggMCBSCi9GMyA5IDAgUgovRjQgMTAgMCBSCi9GNSAxMSAwIFIKPj4KZW5kb2Jq
CjcgMCBvYmoKPDwKL0Jhc2VGb250IC9IZWx2ZXRpY2EKL0VuY29kaW5nIC9XaW5BbnNpRW5jb2RpbmcKL05hbWUgL0YxCi9TdWJ0eXBlIC9UeXBlMQovVHlw
ZSAvRm9udAo+PgplbmRvYmoKOCAwIG9iago8PAovQmFzZUZvbnQgL0hlbHZldGljYS1Cb2xkCi9FbmNvZGluZyAvV2luQW5zaUVuY29kaW5nCi9OYW1lIC9G
MgovU3VidHlwZSAvVHlwZTEKL1R5cGUgL0ZvbnQKPj4KZW5kb2JqCjkgMCBvYmoKPDwKL0Jhc2VGb250IC9IZWx2ZXRpY2EtT2JsaXF1ZQovRW5jb2Rpbmcg
L1dpbkFuc2lFbmNvZGluZwovTmFtZSAvRjMKL1N1YnR5cGUgL1R5cGUxCi9UeXBlIC9Gb250Cj4+CmVuZG9iagoxMCAwIG9iago8PAovQmFzZUZvbnQgL1ph
cGZEaW5nYmF0cwovTmFtZSAvRjQKL1N1YnR5cGUgL1R5cGUxCi9UeXBlIC9Gb250Cj4+CmVuZG9iagoxMSAwIG9iago8PAovQmFzZUZvbnQgL1N5bWJvbAov
TmFtZSAvRjUKL1N1YnR5cGUgL1R5cGUxCi9UeXBlIC9Gb250Cj4+CmVuZG9iagp4cmVmCjAgMTIKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE1IDAw
MDAwIG4gCjAwMDAwMDAwNTQgMDAwMDAgbiAKMDAwMDAwMDExMyAwMDAwMCBuIAowMDAwMDAwMTYyIDAwMDAwIG4gCjAwMDAwMDAzNjEgMDAwMDAgbiAKMDAw
MDAwMzk3OCAwMDAwMCBuIAowMDAwMDA0MDUxIDAwMDAwIG4gCjAwMDAwMDQxNTggMDAwMDAgbiAKMDAwMDAwNDI3MCAwMDAwMCBuIAowMDAwMDA0Mzg1IDAw
MDAwIG4gCjAwMDAwMDQ0NjkgMDAwMDAgbiAKdHJhaWxlcgo8PAovU2l6ZSAxMgovUm9vdCAzIDAgUgovSW5mbyAxIDAgUgo+PgpzdGFydHhyZWYKNDU0Nwol
JUVPRgo=
```

Decode with:

```bash
base64 -d > symbol-encoding-repro.pdf # paste the block above, then Ctrl-D
```

Contributor guide

Open the contributing guide

Research direction

Start with src/v2/pdf_resources/page_font.h, especially get_correct_character() around lines 480–498, then trace the merged textline-cell path that supplies its font. Reproduce with the supplied symbol-encoding-repro.pdf and Python snippet. Done means merged cells decode the Symbol arrow as → and the ZapfDingbats glyph correctly, while the existing unmerged case remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.