firecrawl / firecrawl/pdf-inspector
Two strict-parsing cases reject PDFs that other readers accept (trailing bytes after %%EOF, startxref off by one)
- Dominant language
- Rust
- Stars
- 19.2k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
Moved from firecrawl/anydoc#59 at a maintainer's request.
Reproduced through `firecrawl-anydoc` 0.1.7 (Python binding), which uses pdf-inspector for PDF handling. Linux x86-64, Python 3.10. I have not driven pdf-inspector directly, so the error strings below are the ones anydoc surfaces.
Two independent cases where a PDF is rejected outright although pypdf, pdftotext and browsers all read it without complaint. I hit both in a real corpus: 2 of 12 PDFs failed, so this is not exotic. Both have exact reproductions.
Filing together because they share a theme. The PDF spec is loose about both, and most readers are lenient.
## Case A: more than 506 bytes of trailing data after %%EOF
Error surfaced as `malformed document: invalid PDF structure`.
Some producers pad the file after `%%EOF`. The real file I hit had 80,508 null bytes of padding, produced by iTextSharp 4.1.2. The backward scan for `startxref` looks to be bounded at roughly 507 bytes, so the trailer is never found.
```python
import anydoc
pdf = open('any_valid.pdf','rb').read()
open('a.pdf','wb').write(pdf + b'\x00' * 506)
anydoc.to_markdown('a.pdf') # OK
open('b.pdf','wb').write(pdf + b'\x00' * 507)
anydoc.to_markdown('b.pdf') # fails
```
Bisected precisely: 506 trailing bytes fine, 507 fails. Same result with spaces instead of nulls, so it is a byte-count limit rather than anything about the padding content. pypdf reads all of these.
Suggested fix: scan the whole tail for the last `startxref`, or at minimum use a much larger window. The spec sets no limit on what may follow `%%EOF`.
## Case B: startxref offset points at the EOL immediately before xref
Error surfaced as `couldn't parse input: invalid file trailer`.
Real file, produced by mPDF 8.2.7:
```
startxref
149623
%%EOF
```
The `xref` keyword actually starts at offset 149624. Byte 149623 is the newline before it, so the producer's offset is one byte early. The parser appears to expect `xref` exactly at the given offset and gives up when it finds whitespace.
A PDF from mPDF 8.2.0 in the same corpus parses fine, so this is a producer-version quirk rather than something universal.
```python
import re, anydoc
b = open('any_valid.pdf','rb').read()
m = re.search(rb'startxref\s+(\d+)\s*%%EOF', b)
off = int(m.group(1))
assert b[off-1:off] in (b'\n', b'\r') # EOL sits just before `xref`
patched = b[:m.start(1)] + str(off-1).encode().rjust(len(m.group(1)), b'0') + b[m.end(1):]
open('offby1.pdf','wb').write(patched)
anydoc.to_markdown('offby1.pdf') # fails
```
pypdf reads the patched file and returns the correct page count.
Suggested fix: skip whitespace at the `startxref` offset before expecting the `xref` keyword. Ideally also fall back to scanning for `xref` or `/Type /XRef` when the offset does not resolve, which is what most readers do.
## Why leniency is worth it here
Both cases fail hard with no partial output, so the caller cannot tell a genuinely broken PDF from one with a cosmetic quirk. In a batch pipeline that becomes silent gaps in coverage.
Happy to test a patch.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce both cases through the Python binding entry point anydoc.to_markdown, using the supplied trailing-byte and off-by-one startxref fixtures. Trace the PDF structure and trailer parsing paths in pdf-inspector, then add regression coverage for both malformed-but-readable inputs. Done means both PDFs parse successfully without weakening handling of genuinely invalid documents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100