firecrawl / firecrawl/pdf-inspector
Add RTL/Arabic regression fixtures to CI — generated at test time, nothing to host
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19.2k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
Problem
There is no Arabic or RTL document in the test matrix, and pdf-evals snapshots can't easily cover it — real Arabic documents are mostly copyrighted or private, so there is nothing convenient to host.
That is a problem right now: #212 and PR #207 are both in flight against text_utils.rs, and whichever lands, nothing will catch it silently regressing later. RTL bugs are especially prone to that, because the output looks like text to a reviewer who doesn't read the script — the failure is invisible in a diff.
Proposal: generate the fixtures instead of hosting them
Print HTML to PDF with headless Chrome at test setup. No sample files in the repo, no licensing question, byte-stable across runs, and works in Linux CI:
chrome --headless --disable-gpu --no-pdf-header-footer --print-to-pdf=ar.pdf ar.html
We use exactly this in our own test suite and the same trick is the reproduction in #212, so it is already proven to produce the failing behaviour.
Three assertions worth copying
These are the ones that earned their keep in our suite:
1. Digit groups must survive reordering intact. Assert on a string like 126,248.34 embedded in an Arabic line — it catches digit-run reversal, thousands separators being split, and the decimal point migrating, all in one assertion. This is a live defect: see #212 (25 → 52) and #200.
2. Ligature round-trip. A line containing لا sequences (خلال, السلام, الكلام) must come back character-identical. Multi-character ToUnicode expansions are where naive reversal breaks, and it breaks quietly — the text still reads as Arabic, just wrong.
3. A negative control. This is the one people skip. Run the same input with the RTL path disabled and assert the output is wrong. If a test passes both with and without the fix, it isn't testing the fix — it's testing that the pipeline runs. We caught one of our own tests doing exactly that.
What I can contribute
I can send a PR with:
- a fixture generator (HTML → PDF via headless Chrome, skipping gracefully when Chrome is absent),
- the three assertions above against
extractTextandextractTextWithPositions, - a synthetic corruption generator: apply a fixed letter-substitution map to sound Arabic text to fabricate a "wrong ToUnicode" text layer, so #214-style detection can be tested without hunting for a genuinely corrupt file.
Before I do — two questions so it lands in the right shape:
- Do you want this in this repo's
cargo test, or in thepdf-evalssibling repo alongside the snapshot corpus? - Is a headless-Chrome dependency acceptable in CI, or would you rather I emit the PDFs from a checked-in byte template so the tests have no external dependency at all? The second is more work but keeps CI hermetic — happy either way.
Also relevant: AGENTS.md notes that character-level diff alone misclassifies structural improvements as regressions and that bench.py score is the tie-breaker. For RTL that distinction is sharper than usual — a fix that corrects word order will look like a large diff and score as a regression under a naive comparison. Worth deciding up front which harness these fixtures answer to.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with AGENTS.md and the in-flight text_utils.rs changes, then inspect the existing cargo test and pdf-evals harnesses to decide where these fixtures belong. Compare the headless-Chrome and byte-template options, and verify how extractText, extractTextWithPositions, and bench.py score are used. Done means generated RTL fixtures cover digit groups, ligatures, the negative control, and synthetic corruption without hosted sample files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, ci-cd, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100