firecrawl / firecrawl/pdf-inspector
Valid PDF rejected as `malformed document: invalid PDF structure` when more than 506 trailing bytes follow `%%EOF`
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Summary
pdf-inspector locates the end-of-file marker by scanning only the **last 512
bytes** of the file. Any PDF with more than **506 bytes** of trailing data
after the final `%%EOF` is rejected as `malformed document: invalid PDF
structure`, even though the file is structurally valid and opens correctly in
every other tool.
Trailing data after `%%EOF` is common in the wild: block-padded storage,
some network/streaming writers, and certain producers (notably iTextSharp)
append padding after the EOF marker. These files are handled fine by Acrobat,
poppler (`pdftotext`/`pdfinfo`), and browsers.
## Environment
- Package: `firecrawl-anydoc` **0.1.5** (PyPI wheel `firecrawl_anydoc-0.1.5-cp310-abi3-manylinux_2_17_x86_64`)
- PDF backend: pdf-inspector (via anydoc)
- Python: 3.14.0
- Platform: Linux x86_64
## Steps to reproduce
Self-contained, no external file needed (builds a minimal valid PDF in memory):
```python
import anydoc
def build_pdf() -> bytes:
objs = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
b"<< /Length 44 >>\nstream\n"
b"BT /F1 24 Tf 100 700 Td (Hello World) Tj ET\nendstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
]
out = bytearray(b"%PDF-1.4\n")
offsets = []
for i, body in enumerate(objs, start=1):
offsets.append(len(out))
out += f"{i} 0 obj\n".encode() + body + b"\nendobj\n"
xref_pos = len(out)
out += f"xref\n0 {len(objs)+1}\n".encode() + b"0000000000 65535 f \n"
for off in offsets:
out += f"{off:010d} 00000 n \n".encode()
out += (f"trailer\n<< /Size {len(objs)+1} /Root 1 0 R >>\n"
f"startxref\n{xref_pos}\n%%EOF\n").encode()
return bytes(out)
clean = build_pdf() # 587 bytes, ends with %%EOF\n
for n in (0, 506, 507, 4096, 131072):
try:
anydoc.to_markdown_bytes(clean + b"\x00" * n)
print(f"+{n} trailing bytes: OK")
except Exception as e:
print(f"+{n} trailing bytes: {type(e).__name__}: {e}")
```
### Observed output
```
+0 trailing bytes: OK
+506 trailing bytes: OK
+507 trailing bytes: MALFORMED (ConvertError: malformed document: invalid PDF structure)
+4096 trailing bytes: MALFORMED
+131072 trailing bytes: MALFORMED
```
## Expected behavior
The file converts (or is at least parsed) regardless of trailing padding.
ISO 32000's implementation note has readers locate `%%EOF` within the last
**1024** bytes; mainstream tools go further and tolerate arbitrary trailing
data. pdf-inspector's effective limit is **506 bytes** — half the spec's
guidance and far below real-world padding.
## Actual behavior
`ConvertError: malformed document: invalid PDF structure` once trailing data
exceeds 506 bytes.
## Root cause (measured)
The boundary is exact and points at a fixed 512-byte tail scan. The minimal
PDF is 587 bytes; `%%EOF` begins at offset 581. With `N` trailing bytes the
file is `587 + N` bytes, and `%%EOF` stays inside a trailing 512-byte window
only while `581 >= 587 + N - 512`, i.e. `N <= 506`. The first failure at
`N = 507` matches a parser that reads only the final 512 bytes to find
`%%EOF` / `startxref`.
## Cross-tool check (same padded file)
The 131072-byte-padded variant is handled correctly by poppler:
```
$ pdftotext padded.pdf -
Hello World
$ pdfinfo padded.pdf
Pages: 1
```
Only pdf-inspector rejects it.
## Real-world impact
All laboratory-report PDFs from a major Polish lab chain (Synevo; produced by
iTextSharp, which uses a per-page Form XObject "refry" pattern) carry **4–131
KB** of trailing padding after `%%EOF` (block-padded storage). Every one of
these valid files is rejected as `malformed`, so they cannot be converted at
all. (Reported with a synthetic reproducer because the real files contain
personal medical data.)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the PDF parsing path exercised by anydoc.to_markdown_bytes and reproduce the issue with the supplied in-memory PDF and increasing trailing-byte counts. Trace the EOF-marker scan that currently examines the final 512 bytes; done means the 507-, 4096-, and 131072-byte padded variants parse successfully without regressing the clean file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100