firecrawl / firecrawl/pdf-inspector

Partial CJK corruption on CIDFontType2 subsets whose ToUnicode is a full-range identity bfrange (dompdf)

Open
#246 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.2k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

# Partial CJK corruption on CIDFontType2 subsets whose ToUnicode is a full-range identity `bfrange` (dompdf)

## Summary

For PDFs whose `/ToUnicode` CMap consists of the single degenerate range
`<0000> <0000>`, roughly a third of CJK characters extract as the raw
UTF-16BE bytes of the intended codepoint, split into two Latin-1 characters. The
remaining two thirds decode correctly, so the output looks structurally plausible.

The font is **CIDFontType2** (subset TrueType, Identity-H), so this is not #208 —
that issue covers `try_remap_subset_cmap` misfiring on CIDFontType0/CFF, where the
repair should never run. Here the repair path is the correct one for the font type,
but the source CMap it repairs against is a producer-authored identity assertion
rather than a real mapping, and the result is only partially right.

It is also not caught by the #120 `CipherGarbleStats` discriminator (which resolved
#118). That heuristic is explicitly Latin-dominant — it requires "≥200 ASCII
letters" and keys on vowel ratio and intra-word case transitions. A page that is
mostly correct CJK with ~33% of its ideographs replaced presents no ASCII letter
sample to measure, so the detector cannot fire by construction. `hasEncodingIssues`
stays `false`, `confidence` stays `1.0`, and `pdfType` stays `TextBased`.

## Environment

- `@firecrawl/pdf-inspector` **1.12.0** (npm, prebuilt `darwin-arm64`)
- macOS 15 (Darwin 25.6.0), Apple Silicon
- Producer: **dompdf 3.1.6 + CPDF**, subset CIDFontType2, Identity-H, embedded
Noto Sans HK

## Reproduction

The generator below renders 300 sequential codepoints from U+4E00, one per line, as
plain `

` text — no tables, no headings, no styling — with an embedded
`NotoSansHK-Regular.ttf` (Google Fonts). `isolate.pdf` is attached, along with the
invoice-shaped `repro-cjk-dompdf.pdf`.

```php
set('isRemoteEnabled', true);
$options->set('chroot', '/');
$dompdf = new Dompdf($options);

$lines = '';
for ($i = 0; $i < 300; $i++) {
$ch = mb_chr(0x4E00 + $i, 'UTF-8');
$lines .= "

" . str_pad($i, 3, '0', STR_PAD_LEFT) . " {$ch}

\n";
}

$html = <<
@font-face { font-family:'notosanshk'; font-style:normal; font-weight:normal;
src: url('./fonts/NotoSansHK-Regular.ttf') format('truetype'); }
body { font-family:'notosanshk', sans-serif; font-size: 9pt; }
p { margin: 0; }

{$lines}

HTML;

$dompdf->loadHtml($html, 'UTF-8');
$dompdf->setPaper('A4', 'portrait');
$dompdf->render();
file_put_contents('isolate.pdf', $dompdf->output());
```

```bash
pdf-inspector isolate.pdf
```

**Expected**, and what `pdftotext isolate.pdf -` returns:

```
000 一
001 丁
002 丂
003 七
004 丄
005 丅
006 丆
```

**Actual:**

```
000 一 001 丁 N 003 七 004 丄 005 N 006 N 007 万 008 丈 ...
```

**191 of 286 characters decode correctly; 95 (33%) are corrupted.**

Each corrupted character is emitted as the two bytes of its UTF-16BE form, decoded
as separate Latin-1 characters:

| Index | Expected | Codepoint | Actual |
|---|---|---|---|
| 005 | 丅 | U+4E05 | `N` + U+0005 — `N` is 0x4E, the high byte |
| 035 | 丣 | U+4E23 | `N#` (0x4E, 0x23) |
| 052 | 临 | U+4E34 | `N4` (0x4E, 0x34) |
| 261 | 伅 | U+4F05 | `O` + U+0005 — `O` is 0x4F |

`repro-cjk-dompdf.pdf` is a second, invoice-shaped repro built the same way. There
the corruption is mixed within a single page and a single font: `Customer : Mr.
陳大文` decodes correctly, while a table cell returns
`fz€ýz—|>–ûj_ N[WW‹ U®•‹` for `智能窗簾電機 一字型 單開`. All content in both
files is invented placeholder data.

## The CMap

The `/ToUnicode` stream in these files contains exactly one range:

```
begincmap
...
beginbfrange
<0000> <0000>
endbfrange
endcmap
```

dompdf asserts that every CID equals its Unicode codepoint. For a subset font,
where CIDs are reassigned glyph indices, that is false — but it is well-formed, so
it parses cleanly and looks authoritative.

## Suggested fix

Treat a full-range identity `bfrange` (`<0000> <0000>`) as *absent* rather
than authoritative, and reverse-map glyph IDs through the embedded TrueType `cmap`
table instead. Unlike the file in #118 — which had no `cmap` table and therefore no
in-file source of truth — these fonts embed a complete `cmap`, which is how poppler
recovers the text. `ttf-parser` is already a dependency.

If the decode itself is not changed, this shape at least seems cheaply detectable:
"the only ToUnicode mapping present is a full-range identity" is a structural
signal, not a statistical one, and would let `hasEncodingIssues` fire here. That
would partly serve #122 as well, without needing per-font metadata plumbing.

A CJK-side counterpart to `CipherGarbleStats` may also be worth considering: an
ideograph run interrupted by isolated C0 control characters and stray Latin capitals
is a strong corruption signal, and is what this bug produces every time.

## Impact

The affected files are ordinary business documents — invoices and quotations from a
dompdf-based system. Across 22 such files, essentially all Chinese was lost, worst
case 9 of 1219 characters recovered, with no error raised and full reported
confidence. Latin text in the same documents extracts perfectly, so nothing looks
wrong at a glance.

Everything else about the library has been excellent — classification accuracy and
speed are as advertised, and non-dompdf CJK PDFs (SimSun, Microsoft YaHei, Type 3
Noto CJK) extract flawlessly. Thanks for open-sourcing it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the try_remap_subset_cmap repair path and inspect how ttf-parser exposes the embedded TrueType cmap. Reproduce with the attached isolate.pdf using pdf-inspector, then verify that a full-range identity ToUnicode CMap is handled through the embedded cmap or reliably flagged as an encoding issue, without regressing the CIDFontType0/CFF behavior from #208.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.