firecrawl / firecrawl/pdf-inspector
detector: text-ratio branch shadows the Mixed classification when scanned pages carry a little real text
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
In `detect_from_document` (src/detector.rs), the classification branches are ordered like this:
```rust
let (pdf_type, confidence) = if has_template_images && pages_with_text > 0 {
(PdfType::Mixed, ...) // ocr_recommended = true
} else if text_ratio >= config.text_page_ratio_threshold {
(PdfType::TextBased, text_ratio) // ocr_recommended = false
} else if pages_with_text == 0 && (pages_with_images > 0 || ...) {
(PdfType::Scanned / ImageBased, ...)
} else if pages_with_text > 0 && (pages_with_images > 0 || ...) {
(PdfType::Mixed, 0.7) // rarely reachable
} else ...
```
The `text_ratio` arm is evaluated before the "has text AND has images" arm, so that Mixed arm is only reachable when `text_ratio` is *below* the threshold. A document where every page passes the text-page test short-circuits to `TextBased` (`ocr_recommended = false`, `pages_needing_ocr` empty) no matter how image-heavy the pages are. The only escape hatch is `has_template_images`, which is deliberately narrow (single image, `text_operator_count < 50`, low alphanumeric diversity).
Concrete case that would misroute: a scanned form where each page is one full-page raster plus a real digitally-generated header or footer. With images present the effective minimum is 10 text ops; a normal header/footer clears that, so every page counts as a text page, `text_ratio = 1.0`, and the document classifies `TextBased` with the scan content silently dropped. Because `TextBased` skips the per-page OCR pass entirely, the miss is total rather than partial.
Config can't work around it: raising `min_text_ops_per_page` only flips pages to "no text" (routing to Scanned/ImageBased for the whole document), and `text_page_ratio_threshold` can't distinguish "text page" from "text page that also carries a full-page image".
To be fair, defaults classified everything in my test set correctly — the template-image heuristic covers the common scan-with-OCR-overlay shape, and this only bites when the per-page text is real but incidental. Filing it as a latent ordering issue: one option is to gate the `text_ratio` short-circuit on image content, e.g. fall through to the Mixed arm when a large share of the text pages also carry near-full-page images.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/detector.rs at detect_from_document and trace how text_ratio, pages_with_text, pages_with_images, and has_template_images determine PdfType and OCR metadata. Reproduce the scanned-form case described in the issue, then add coverage showing that image-heavy pages with incidental real text are not classified as purely TextBased and that the resulting OCR routing is correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100