firecrawl / firecrawl/pdf-inspector
Page-number stripping deletes body content on A4 pages (PAGE_NUMBER_TOP_Y is US-Letter-calibrated)
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Summary
`page_number_value` treats any short all-digit text item near the top or bottom of a page as a page-number candidate, using the **absolute** constant `PAGE_NUMBER_TOP_Y = 720.0`. That constant is never scaled by the actual page height.
On US Letter (792 pt) the top band is 72 pt — one inch, as intended. On **A4 (841.89 pt) it is the top 121.89 pt, roughly 14.5% of the page**, or about eight lines of body text — even though the comment beside the constant says "top ~5%".
The result is that body content gets silently deleted from A4 documents. The output stays well-formed, so the loss is easy to miss: a table keeps a valid pipe structure and simply loses a column.
## Minimal reproduction
Self-contained, no external files. Same two rows, drawn twice on an A4 page — once at y=740/725, once at y=700/685:
```python
"""Minimal repro: same two rows, once at y=740 and once at y=700, on A4."""
import pdf_inspector as pi
def make_pdf(path, rows):
lines = []
for y, idx, txt in rows:
lines.append(f"BT /F1 9 Tf 208 {y} Td ({idx}) Tj ET")
lines.append(f"BT /F1 9 Tf 226 {y} Td ({txt}) Tj ET")
content = "\n".join(lines).encode()
objs = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595.28 841.89] "
b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
b"<< /Length %d >>\nstream\n" % len(content) + content + b"\nendstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
]
out = bytearray(b"%PDF-1.4\n"); offs = []
for i, o in enumerate(objs, 1):
offs.append(len(out)); out += b"%d 0 obj\n" % i + o + b"\nendobj\n"
x = len(out)
out += b"xref\n0 %d\n0000000000 65535 f \n" % (len(objs) + 1)
for o in offs: out += b"%010d 00000 n \n" % o
out += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (len(objs) + 1, x)
open(path, "wb").write(bytes(out))
for label, base in (("TOP (y=740/725)", 740), ("BODY (y=700/685)", 700)):
make_pdf("r.pdf", [(base, "1", "22 December 2026"), (base - 15, "2", "22 January 2027")])
print(f"{label}: {pi.process_pdf('r.pdf').markdown.strip()!r}")
```
Output:
```
TOP (y=740/725): '22 December 2026 22 January 2027'
BODY (y=700/685): '1 22 December 2026 2 22 January 2027'
```
The indices `1` and `2` are deleted in the first case and kept in the second. Only the y offset differs. A 1 pt bisection on a real document puts the boundary exactly at 720: at `y = 720.78` the index is lost, at `y = 719.78` it survives. Changing the MediaBox height does not move the boundary, confirming the threshold is absolute.
## Root cause
`src/extractor/layout.rs:973-995`:
```rust
const PAGE_NUMBER_BOTTOM_Y: f32 = 100.0;
const PAGE_NUMBER_TOP_Y: f32 = 720.0;
fn page_number_value(item: &TextItem) -> Option {
// ...
let text = item.text.trim();
if text.is_empty() || text.len() > 4 || !text.chars().all(|c| c.is_ascii_digit()) {
return None;
}
// Must be at top or bottom of page.
// US Letter = 792pt, A4 = 841pt. Page numbers are typically in the
// top ~5% or bottom ~12% of the page.
if item.y <= PAGE_NUMBER_TOP_Y && item.y >= PAGE_NUMBER_BOTTOM_Y {
return None;
}
text.parse().ok()
}
```
Removal happens in `page_number_removal_mask` (`src/extractor/layout.rs:1605-1615`) with no cross-page corroboration:
```rust
.map(|(index, value)| explicit_folio[index] || (value.is_some() && !contextual[index]))
```
The predicate is only "1–4 ASCII digits", so the magnitude is unbounded: on a real document, shifting content upward progressively deletes every index that crosses y=720, including two-digit ones (`10`…`18`).
The `contextual` guard does not appear to engage in this layout. With the synthetic PDF above, a digit flanked by words on the same baseline is still deleted (`Row 1 December` → `Row December`) at every column spacing tested from 8 pt to 24 pt, while the identical row at y=700 is kept.
## The tokens are present — the loss is in markdown assembly
Two independent extractors recover them from the same file:
```
pdftotext (poppler): n=\n1\n\n22 December 2026\n\n2\n\n22 January 2027 …
pypdf: 'n = ' / '1 22 December 2026 ' / '2 22 January 2027 ' …
```
and `extract_text_with_positions` still reports them with correct coordinates:
```
y=753.78 x=208.4 'n = '
y=738.78 x=208.4 '1 ' x=226.4 '22 December 2026 '
y=723.78 x=208.4 '2 ' x=226.4 '22 January 2027 '
y=708.78 x=208.4 '3 ' x=226.4 '22 February 2027 ' <- survives
```
So the text extraction is fine; the deletion is introduced only on the markdown path.
## Real-world impact
We parse A4 financial documents (EU securities Final Terms and PRIIPs KIDs). Across a 17-document corpus from six issuers, digit-only tokens above y=720 were deleted in six documents. Excluding genuine folios below y=100, the losses were:
| Document | Page | Deleted | What it was |
|---|---|---|---|
| Final Terms A (29 pp) | 20 | `13 14 15 28 29 30` | row indices of an autocall payment schedule |
| " | 17, 24 | `2025 2024 2023` | year column headers in the financial statements |
| Final Terms B (29 pp) | 9 | `4` | a row index in an underlying table |
| KID (3 pp) | 2 | `5` | a step of the 1–7 risk scale |
| Final Terms C (10 pp) | 3, 6 | `1 2` (each) | row indices of two redemption schedules |
The dangerous property is that the emitted markdown is still **structurally valid**:
```
|10 June 2027|87.00%|100.00%|17 June 2027| <- index 14 gone
|12 July 2027|86.00%|100.00%|19 July 2027| <- index 15 gone
```
Row count, column count and date ordering all pass. Nothing short of an explicit "indices must be contiguous 1..N" invariant detects it, and that invariant is only writable by someone who already knows about this behaviour.
Whether a given document is affected depends purely on where its table happens to land on the page, so a document set that looks clean today can start losing rows after an unrelated layout change upstream.
## No way to turn it off
`filter_markdown_page_numbers_with_removed_pages` is called unconditionally (`src/lib.rs:489-490`). `MarkdownOptions::include_page_numbers` does not control it — it only decides whether `` comments are emitted (`src/markdown/convert.rs:855`, `:1310`). The Python binding is `process_pdf(path, pages)` and exposes no options at all, so from Python the behaviour cannot be influenced.
## Suggested fixes
1. **Scale the band by the page's actual MediaBox height** (top 5% / bottom 12%, matching the comment) instead of the absolute 720.0 / 100.0 constants. This is the correct fix but appears to need page geometry threaded into `layout.rs`, since `TextItem` carries no page dimensions.
2. **A smaller fix that needs no new data: require a page-number candidate to be alone on its baseline.** A real folio is typically the only item on its line; in every case above the deleted digit shared a baseline with ordinary body text one column to the right. Rejecting candidates that have a neighbour on the same baseline within the existing `PAGE_NUMBER_Y_TOLERANCE` would eliminate these false positives. Requiring corroboration across pages (same x/y, advancing value) before deletion would harden it further.
Even without a fix, exposing the removal as an option on `PdfOptions` and on the Python bindings would let callers opt out.
## Environment
- `pdf-inspector` 0.2.6 (PyPI wheel), Python 3.12.13, macOS 26.5.1 (arm64)
- Source read at `585d36e` ("fix: recover from a corrupted startxref pointer (#230)")
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the minimal A4 reproduction and inspect page_number_value in src/extractor/layout.rs:973-995, then trace page_number_removal_mask at lines 1605-1615. Check the unconditional call in src/lib.rs:489-490 and the MarkdownOptions behavior in src/markdown/convert.rs. Done means body digits near the top of A4 pages are retained while genuine page-number removal still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100