firecrawl / firecrawl/pdf-inspector

Expose signals the detector already computes: reading direction and per-page confidence

Open
#217 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Summary

Two signals are computed inside the library and discarded before the public API. Both are cheap to surface, neither changes behaviour, and both would have turned recent bug reports into self-diagnosis.

This follows the same pattern as #36 ("the detector computes `PageAnalysis.has_images` then discards it before the public API").

## 1. Reading direction

`is_rtl_text` already decides whether the rightmost column is read first — it drives `src/extractor/reading_order.rs`. But `is_rtl_char` and `is_rtl_text` are `pub(crate)`, and nothing in `PdfResult`, `PageMarkdownResult` or `TextItem` says which direction was applied.

Grepping the public surface: `src/lib.rs` contains the string `rtl` zero times, and `language` zero times.

**Why it matters:** when RTL handling misfires (#212), the caller sees plausible-looking text and no indication that anything script-specific happened. There is no way to answer "did the RTL path fire on this page?" from the outside, so a silent wrong answer is indistinguishable from a correct one. A `direction: "ltr" | "rtl" | "mixed"` on the page result, or `isRtl` on `TextItem`, makes it self-diagnosing.

It is also useful in its own right: a caller rendering extracted text needs to know the base direction, and today has to re-derive it by inspecting code points.

## 2. Per-page confidence

`PdfClassification` exposes one `confidence: number` for the whole document, alongside a per-page `pagesNeedingOcr` array. So the caller learns *which* pages were flagged but not *how strongly*, and cannot see which pages dragged the document score down.

In #213 the result was `Mixed`, confidence 0.7, 24 of 25 pages flagged — with no way to tell whether those 24 pages were marginal or emphatic, or whether one page was pathological and the rest borderline. Diagnosing it required a full comparison run against poppler. `perPageConfidence: number[]` would have made it a one-line report.

## 3. While you're there: document the `ocrReason` vocabulary

`ocrReason` is typed as a bare `string`, and the vocabulary appears to be four literals — `no_text`, `suspected_garbled_text`, `vector_text`, `phantom_empty_row`. Callers routing on these are string-matching against an undocumented, unstable set. A union type (or a documented list) costs nothing and makes the routing contract explicit. Happy to send that as a small PR if useful.

## Why these three together

The library's product thesis is routing: decide locally whether a document needs an expensive OCR service. Every one of these is about making that decision *auditable* by the caller. Right now a wrong routing decision is invisible until someone compares against another extractor — which is how both #212 and #213 were found.

None of these change any behaviour, so they should be safe to land independently of the fixes they'd help diagnose.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/lib.rs and the public PdfResult, PageMarkdownResult, TextItem, and PdfClassification types, then trace direction handling in src/extractor/reading_order.rs. Identify where the computed RTL signal, per-page confidence, and ocrReason values are discarded or exposed, and verify that the public API reports them without changing extraction or classification behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.