firecrawl / firecrawl/pdf-inspector

Feature: report bench scores per document type, not one composite

Open
#503 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Problem

`CLAUDE.md` makes `bench.py score` over `pdf-evals` the semantic verdict, and warns that a character-level diff alone misclassifies structural improvements — a column-detection rewrite, say — as regressions.

A single composite over a mixed corpus has that same failure mode one level up. Document classes do not move together: a change that improves scanned newspapers can cost a point on clean single-column articles, and the composite reports one number that describes neither.

The spread between classes is not small. The Unlimited-OCR paper (arXiv 2606.23050) reports roughly a tenfold difference across document types for one model — about 0.012 on research papers against 0.134 on newspapers. A corpus mixing those classes reports an average that describes no document in it.

## Why this repo specifically

Two current workstreams are exactly where a composite misleads:

- **RTL ordering** (#212 and the PRs around it). A correct fix rewrites the character order of every Arabic document in the corpus. Against a snapshot baseline that reads as a large regression on those files and a no-op everywhere else, so the composite averages a real fix into noise — the same trap `CLAUDE.md` already warns about, one level up.
- **Table and layout recovery.** Structural wins concentrate in a few classes and disappear in the mean.

## Proposal

1. A `type` label per corpus entry — a plain string in the snapshot metadata (`article`, `book`, `form`, `invoice`, `newspaper`, `scan`, `rtl`, …). No new dependency, no model, no change to scoring itself.
2. `bench.py score` prints a per-type table above the composite; `bench.py test` reports which types changed.
3. Optionally, a flag to fail only when a *type* regresses past a threshold, so a deliberate rewrite in one class need not be argued against a corpus mean.

The composite stays. It just stops being the only view.

I can supply the labelling rule and a starting label set for the Arabic and RTL portion of the corpus if that helps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading CLAUDE.md, then locate bench.py and its score and test entry points, along with the snapshot metadata for pdf-evals. Define how the proposed type labels feed the per-type table and changed-type reporting while preserving the composite; done means both commands expose type-level results, with threshold behavior resolved if included.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.