firecrawl / firecrawl/pdf-inspector
extract_pages_markdown flags a native text page as suspected_garbled_text (empty markdown) while classify_pdf / detect_pdf call it text_based; other extractors read the page fine
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
Versions: pdf-inspector 1.19.0 (PyPI, Python 3.12.10, Windows 11), reached through firecrawl-anydoc 0.2.4, where it surfaces as NeedsOcrError: page 2 of 57 needs OCR and, under the default ocr="reject", refuses the whole document. anydoc 0.2.3 converted the same page correctly.
The page
Page 2 of a 57 page sell side research PDF: the table of contents. Native text, nothing raster:
Content stream: 66 BT … ET text objects (49 TJ, 17 Tj), two small vector fills, no Do operator. Seven image XObjects are declared in the shared /Resources but never painted.
Fonts: subsetted TrueType (Frutiger 45 Light / Bold, UBSLogo), /Subtype /TrueType, no /Encoding, symbolic flag set, /ToUnicode present. Pages 1 and 3 use the same font resources.
pdftotext (poppler) extracts 1,476 characters from the page; PyMuPDF 1,504; pypdf 1,499. anydoc 0.2.3 produced ### Contents followed by the entries with dot leaders and page numbers.
What pdf-inspector 1.19.0 does with it
On the page extracted into a single page PDF (and identically on the full document, where it is page 2):
classify_pdf(path) -> pdf_type='text_based', confidence=1.0, pages_needing_ocr=[]
detect_pdf(path) -> pdf_type='text_based', pages_needing_ocr=[], has_encoding_issues=False
extract_pages_markdown(path) -> pages_needing_ocr=[1]
ocr_reasons_by_page=[PageOcrReasons(page=1, reasons=['suspected_garbled_text'])]
PageMarkdown(page=0, markdown='', needs_ocr=True)
extract_text(path) -> 637 characters, 11% letters: '1! \n … !"#$%$ \n … &\nÆÆ(Æ)Æ# \n … *\n+ÆÆ,'
So the fast classifier and the detector both say text based with no OCR pages, but the markdown path returns nothing for the page and reports it as garbled. The plain extract_text output looks like the decoded glyph codes of the symbolic fonts emitted as characters rather than mapped through /ToUnicode, which would explain why the garbled text heuristic fires: it is judging pdf-inspector's own decoding, not the page. Pages 1 and 3 of the same document, with the same font resources, come through the markdown path fine, so whatever differs is specific to this page's text objects; I could not isolate it from outside (removing the vector fills, the marked content wrappers, or moving Tf inside the text objects all leave the result unchanged, and synthetic contents pages with dot leaders convert normally).
Impact through anydoc
Under anydoc 0.2.4 one page flagged this way refuses all 57 pages, and there is no mode that returns the other 56. On my corpus of 67 research PDFs, 0.2.4 refuses three documents this way, all of which 0.2.3 converted. I have worked around it by catching NeedsOcrError.pages, dropping those pages and converting the rest, which recovers the 0.2.3 output exactly, but the page itself is lost.
Requests
A look at the markdown path's text decoding for symbolic TrueType fonts carrying /ToUnicode, on this page. I can share the single page PDF privately (it is licensed research content, so not attached here); it reproduces on its own.
Consistency between classify_pdf / detect_pdf and extract_pages_markdown: when the first two say text based and the third says OCR, callers such as anydoc have no way to tell a scan from a decoding failure.
In anydoc, an ocr="skip" style mode that returns the readable pages and lists the flagged ones, so a single flagged page does not discard a long document. I will file that part on the anydoc tracker if you prefer it there.
Thanks for both libraries; the table rendering is why anydoc is my primary extractor.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at extract_pages_markdown, extract_text, classify_pdf, and detect_pdf, then reproduce with the single-page PDF when it is available. Compare symbolic TrueType decoding and the suspected_garbled_text decision for this page against pages 1 and 3. Done means the native text page is not incorrectly routed to OCR and the classification results are consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100