firecrawl / firecrawl/pdf-inspector
Partially garbled OCR pages report high confidence, empty warnings, and no hosted recommendation: the page-level mean cannot express a bad region
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
### Summary
A page whose OCR output is partly garbage comes back with `ocr_confidence` high, `warnings` empty, and `hosted_recommended` false. The page-level mean is the only quality signal that reaches the caller, and a mean answers *"is this page mostly fine?"* while an integrator needs *"is any part of this page unusable?"*. Those two questions diverge exactly on the common real-world page: body text plus a small dense region (chart axis, legend, footnote strip, stamp).
The result is silent: garbled characters sit inline with good prose, at the same reported confidence, with nothing marking them.
### Reproduction
Synthetic page, one page, image-only (no text layer — `pdftotext` returns zero characters), composed of two rasterized regions:
- body text at normal size: `Sample readings by bucket` / `The series below is a routine measurement table.`
- a strip of tick labels reduced to roughly 3pt: `0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230`
```js
const { processPdfWithOcr } = require('@firecrawl/pdf-inspector');
const r = await processPdfWithOcr(readFileSync('mixed-quality.pdf'), { mode: 'Auto' });
console.log(r.markdown);
console.log(r.pagesRecommendingHosted, r.pages[0].provenance.ocrConfidence, r.pages[0].provenance.warnings);
```
**Observed**
```
Sample readings by bucket The series below is a routine measurement table.
## 0230203040506070809010011012013010150160170 10190200210220
```
```
pagesRecommendingHosted: []
ocrConfidence: 0.966
warnings: []
```
The body text is transcribed perfectly. The strip loses every separator, drops `140` and `180`, and pulls the wrapped `230` up to position 2 — so the region is not merely unformatted, its values are wrong. It is then promoted to an `##` heading. Nothing in the result distinguishes those characters from the correct line above them.
Same shape on a real document: a page of study notes with an embedded chart returned the axis as one glued digit run and rendered the legend `SD = 10 / SD = 50` as `SD=110`, again with `warnings: []`, `pagesRecommendingHosted: []`, and page confidence 0.95.
### Why it happens
Reading `main` at 1.17.0:
- `src/vision/contracts.rs:144` — every recognition span carries its own `confidence`.
- `src/vision/contracts.rs:156` and `:201` — the page keeps only `mean_confidence` / `ocr_confidence`, *"mean across accepted spans"*. The distribution is discarded at that boundary.
- `src/vision/contracts.rs:38,49` — `minimum_confidence` defaults to `0.0`, so by default no weak span is dropped; the weak spans stay in the output *and* in the mean.
- `src/vision/pipeline.rs:81,92` — `hosted_recommendation_confidence` defaults to `0.5`, compared against that page-level value.
- `src/vision/pipeline.rs:447-450` — `pages_recommending_hosted` filters on `provenance.hosted_recommended`.
Twenty bad spans among two hundred good ones cannot move a mean below `0.5`, so the threshold can never fire on a partially bad page. The signal is structurally unable to express the failure it is meant to catch. `docs/ocr-runtime.md` states that `pages_recommending_hosted` marks pages that are *"empty, low-confidence, or still appear incomplete"* — a partially garbled page is none of the three by page-level mean, yet it is the case where an integrator most needs the warning.
This is the OCR-side twin of #352, where a page-level `CipherGarbleStats` signal is bypassed by a page that mixes healthy and garbled fonts. Same architecture, different pipeline: a per-page aggregate cannot describe a sub-page defect.
### Suggested fix
Both parts are additive and backward compatible; the span confidences already exist, nothing new has to be computed.
1. **Keep the shape of the distribution, not just its centre.** In `PageProvenance` (`src/vision/contracts.rs`, next to `ocr_confidence`), add:
```rust
/// Lowest accepted span confidence on this page.
pub min_confidence: Option,
/// Accepted spans below the hosted-recommendation threshold.
pub low_confidence_spans: u32,
```
A caller can then flag the page — or the region — without guessing. Today there is no field that could carry this information.
2. **Make the hosted recommendation a worst-region test in OR with the mean.** In the routing decision behind `hosted_recommended` (`src/vision/pipeline.rs`), recommend hosted when a contiguous run of spans below `hosted_recommendation_confidence` exceeds a small character budget, in addition to the existing mean comparison. A run of adjacent weak spans is exactly the signature of one bad region on an otherwise good page, and it is invisible to any average.
An alternative that needs no new field: change the default `minimum_confidence` from `0.0` to a non-zero value so weak spans are dropped rather than emitted. I would argue against it on its own — dropping is also silent, and a missing region is harder to notice than a wrong one — but combined with (1) it would at least keep known-bad characters out of the markdown.
### Environment
`@firecrawl/pdf-inspector` 1.17.0, Node bindings, macOS arm64, PDFium `native-v7988`, ONNX Runtime 1.27.0, `mode: 'Auto'`, defaults otherwise. The recogniser quality is not the complaint here and may well be platform-dependent — the reporting path is the same on every platform, and that is what this issue is about.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the span and page provenance definitions in src/vision/contracts.rs, then trace hosted_recommended and pages_recommending_hosted in src/vision/pipeline.rs. Reproduce the mixed-quality case through processPdfWithOcr and compare the result with docs/ocr-runtime.md; done means partial low-confidence regions are represented and the routing decision can flag them without breaking existing aggregate fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, rust
- Domain
- backend-api-design, computer-vision
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100