firecrawl / firecrawl/pdf-inspector
Feature: report bench scores per document type, not one composite
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Problem
`CLAUDE.md` makes `bench.py score` over `pdf-evals` the semantic verdict, and warns that a character-level diff alone misclassifies structural improvements — a column-detection rewrite, say — as regressions.
A single composite over a mixed corpus has that same failure mode one level up. Document classes do not move together: a change that improves scanned newspapers can cost a point on clean single-column articles, and the composite reports one number that describes neither.
The spread between classes is not small. The Unlimited-OCR paper (arXiv 2606.23050) reports roughly a tenfold difference across document types for one model — about 0.012 on research papers against 0.134 on newspapers. A corpus mixing those classes reports an average that describes no document in it.
## Why this repo specifically
Two current workstreams are exactly where a composite misleads:
- **RTL ordering** (#212 and the PRs around it). A correct fix rewrites the character order of every Arabic document in the corpus. Against a snapshot baseline that reads as a large regression on those files and a no-op everywhere else, so the composite averages a real fix into noise — the same trap `CLAUDE.md` already warns about, one level up.
- **Table and layout recovery.** Structural wins concentrate in a few classes and disappear in the mean.
## Proposal
1. A `type` label per corpus entry — a plain string in the snapshot metadata (`article`, `book`, `form`, `invoice`, `newspaper`, `scan`, `rtl`, …). No new dependency, no model, no change to scoring itself.
2. `bench.py score` prints a per-type table above the composite; `bench.py test` reports which types changed.
3. Optionally, a flag to fail only when a *type* regresses past a threshold, so a deliberate rewrite in one class need not be argued against a corpus mean.
The composite stays. It just stops being the only view.
I can supply the labelling rule and a starting label set for the Arabic and RTL portion of the corpus if that helps.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading CLAUDE.md, then locate bench.py and its score and test entry points, along with the snapshot metadata for pdf-evals. Define how the proposed type labels feed the per-type table and changed-type reporting while preserving the composite; done means both commands expose type-level results, with threshold behavior resolved if included.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100