firecrawl / firecrawl/pdf-inspector

Valid PDF rejected as `malformed document: invalid PDF structure` when more than 506 trailing bytes follow `%%EOF`

Open
#286 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Summary

pdf-inspector locates the end-of-file marker by scanning only the **last 512
bytes** of the file. Any PDF with more than **506 bytes** of trailing data
after the final `%%EOF` is rejected as `malformed document: invalid PDF
structure`, even though the file is structurally valid and opens correctly in
every other tool.

Trailing data after `%%EOF` is common in the wild: block-padded storage,
some network/streaming writers, and certain producers (notably iTextSharp)
append padding after the EOF marker. These files are handled fine by Acrobat,
poppler (`pdftotext`/`pdfinfo`), and browsers.

## Environment

- Package: `firecrawl-anydoc` **0.1.5** (PyPI wheel `firecrawl_anydoc-0.1.5-cp310-abi3-manylinux_2_17_x86_64`)
- PDF backend: pdf-inspector (via anydoc)
- Python: 3.14.0
- Platform: Linux x86_64

## Steps to reproduce

Self-contained, no external file needed (builds a minimal valid PDF in memory):

```python
import anydoc

def build_pdf() -> bytes:
objs = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
b"<< /Length 44 >>\nstream\n"
b"BT /F1 24 Tf 100 700 Td (Hello World) Tj ET\nendstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
]
out = bytearray(b"%PDF-1.4\n")
offsets = []
for i, body in enumerate(objs, start=1):
offsets.append(len(out))
out += f"{i} 0 obj\n".encode() + body + b"\nendobj\n"
xref_pos = len(out)
out += f"xref\n0 {len(objs)+1}\n".encode() + b"0000000000 65535 f \n"
for off in offsets:
out += f"{off:010d} 00000 n \n".encode()
out += (f"trailer\n<< /Size {len(objs)+1} /Root 1 0 R >>\n"
f"startxref\n{xref_pos}\n%%EOF\n").encode()
return bytes(out)

clean = build_pdf() # 587 bytes, ends with %%EOF\n
for n in (0, 506, 507, 4096, 131072):
try:
anydoc.to_markdown_bytes(clean + b"\x00" * n)
print(f"+{n} trailing bytes: OK")
except Exception as e:
print(f"+{n} trailing bytes: {type(e).__name__}: {e}")
```

### Observed output

```
+0 trailing bytes: OK
+506 trailing bytes: OK
+507 trailing bytes: MALFORMED (ConvertError: malformed document: invalid PDF structure)
+4096 trailing bytes: MALFORMED
+131072 trailing bytes: MALFORMED
```

## Expected behavior

The file converts (or is at least parsed) regardless of trailing padding.
ISO 32000's implementation note has readers locate `%%EOF` within the last
**1024** bytes; mainstream tools go further and tolerate arbitrary trailing
data. pdf-inspector's effective limit is **506 bytes** — half the spec's
guidance and far below real-world padding.

## Actual behavior

`ConvertError: malformed document: invalid PDF structure` once trailing data
exceeds 506 bytes.

## Root cause (measured)

The boundary is exact and points at a fixed 512-byte tail scan. The minimal
PDF is 587 bytes; `%%EOF` begins at offset 581. With `N` trailing bytes the
file is `587 + N` bytes, and `%%EOF` stays inside a trailing 512-byte window
only while `581 >= 587 + N - 512`, i.e. `N <= 506`. The first failure at
`N = 507` matches a parser that reads only the final 512 bytes to find
`%%EOF` / `startxref`.

## Cross-tool check (same padded file)

The 131072-byte-padded variant is handled correctly by poppler:

```
$ pdftotext padded.pdf -
Hello World
$ pdfinfo padded.pdf
Pages: 1
```

Only pdf-inspector rejects it.

## Real-world impact

All laboratory-report PDFs from a major Polish lab chain (Synevo; produced by
iTextSharp, which uses a per-page Form XObject "refry" pattern) carry **4–131
KB** of trailing padding after `%%EOF` (block-padded storage). Every one of
these valid files is rejected as `malformed`, so they cannot be converted at
all. (Reported with a synthetic reproducer because the real files contain
personal medical data.)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the PDF parsing path exercised by anydoc.to_markdown_bytes and reproduce the issue with the supplied in-memory PDF and increasing trailing-byte counts. Trace the EOF-marker scan that currently examines the final 512 bytes; done means the 507-, 4096-, and 131072-byte padded variants parse successfully without regressing the clean file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.