firecrawl / firecrawl/pdf-inspector

Two strict-parsing cases reject PDFs that other readers accept (trailing bytes after %%EOF, startxref off by one)

Open
#301 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.2k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

Moved from firecrawl/anydoc#59 at a maintainer's request.

Reproduced through `firecrawl-anydoc` 0.1.7 (Python binding), which uses pdf-inspector for PDF handling. Linux x86-64, Python 3.10. I have not driven pdf-inspector directly, so the error strings below are the ones anydoc surfaces.

Two independent cases where a PDF is rejected outright although pypdf, pdftotext and browsers all read it without complaint. I hit both in a real corpus: 2 of 12 PDFs failed, so this is not exotic. Both have exact reproductions.

Filing together because they share a theme. The PDF spec is loose about both, and most readers are lenient.

## Case A: more than 506 bytes of trailing data after %%EOF

Error surfaced as `malformed document: invalid PDF structure`.

Some producers pad the file after `%%EOF`. The real file I hit had 80,508 null bytes of padding, produced by iTextSharp 4.1.2. The backward scan for `startxref` looks to be bounded at roughly 507 bytes, so the trailer is never found.

```python
import anydoc
pdf = open('any_valid.pdf','rb').read()

open('a.pdf','wb').write(pdf + b'\x00' * 506)
anydoc.to_markdown('a.pdf') # OK

open('b.pdf','wb').write(pdf + b'\x00' * 507)
anydoc.to_markdown('b.pdf') # fails
```

Bisected precisely: 506 trailing bytes fine, 507 fails. Same result with spaces instead of nulls, so it is a byte-count limit rather than anything about the padding content. pypdf reads all of these.

Suggested fix: scan the whole tail for the last `startxref`, or at minimum use a much larger window. The spec sets no limit on what may follow `%%EOF`.

## Case B: startxref offset points at the EOL immediately before xref

Error surfaced as `couldn't parse input: invalid file trailer`.

Real file, produced by mPDF 8.2.7:

```
startxref
149623
%%EOF
```

The `xref` keyword actually starts at offset 149624. Byte 149623 is the newline before it, so the producer's offset is one byte early. The parser appears to expect `xref` exactly at the given offset and gives up when it finds whitespace.

A PDF from mPDF 8.2.0 in the same corpus parses fine, so this is a producer-version quirk rather than something universal.

```python
import re, anydoc
b = open('any_valid.pdf','rb').read()
m = re.search(rb'startxref\s+(\d+)\s*%%EOF', b)
off = int(m.group(1))
assert b[off-1:off] in (b'\n', b'\r') # EOL sits just before `xref`
patched = b[:m.start(1)] + str(off-1).encode().rjust(len(m.group(1)), b'0') + b[m.end(1):]
open('offby1.pdf','wb').write(patched)

anydoc.to_markdown('offby1.pdf') # fails
```

pypdf reads the patched file and returns the correct page count.

Suggested fix: skip whitespace at the `startxref` offset before expecting the `xref` keyword. Ideally also fall back to scanning for `xref` or `/Type /XRef` when the offset does not resolve, which is what most readers do.

## Why leniency is worth it here

Both cases fail hard with no partial output, so the caller cannot tell a genuinely broken PDF from one with a cosmetic quirk. In a batch pipeline that becomes silent gaps in coverage.

Happy to test a patch.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce both cases through the Python binding entry point anydoc.to_markdown, using the supplied trailing-byte and off-by-one startxref fixtures. Trace the PDF structure and trailer parsing paths in pdf-inspector, then add regression coverage for both malformed-but-readable inputs. Done means both PDFs parse successfully without weakening handling of genuinely invalid documents.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.