firecrawl / firecrawl/pdf-inspector
Feature request: report removed header/footer lines (audit trail for furniture stripping)
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Problem
`strip_headers_footers` (and the folio filtering) remove text with no trace of what or how much was removed. For ingestion pipelines that persist extracted text (RAG, archival), silent removal is hard to operate: when a customer asks "where did my footer text go" — or worse, when the furniture heuristic misfires on a boundary case — there is nothing to audit.
Our pipeline stamps every page that lost lines with a removed-count in page metadata, so removals are visible without diffing outputs. We currently do this with a layout-model-based extractor and would like parity when processing via pdf-inspector.
## Proposal
Add a per-page removed-line count to `PdfProcessResult`, e.g.:
```rust
/// 1-indexed page → number of text lines removed as running headers/footers.
pub removed_header_footer_lines: Vec<(u32, u32)>,
```
Sketch (happy to implement if the direction sounds right):
- Compute per-page counts at the existing gate in `markdown/mod.rs` (diff the per-page line counts around `furniture::strip_header_footer_lines`) — no change to `furniture.rs` itself.
- Thread them through `MarkdownConversionOutput` (the struct already exists for `markdown` + `detected_tables`).
- Surface on `PdfProcessResult` and the Python/Node/WASM result wrappers (or Python-only to start, if you prefer keeping the other bindings lean).
Open questions where I'd take your steer:
1. Total vs per-page (per-page is what an audit trail needs; total is cheaper to plumb).
2. Whether folio removals (`select_items_with_document_folio_context`) should be counted in the same field or reported separately — they are removed unconditionally, so an audit entry is arguably even more valuable there.
3. Whether `MarkdownConversionOutput` is the right vehicle or you'd rather grow the `MarkdownDocumentContext` with an out-slot.
Related: the sibling PR exposing `strip_headers_footers` / `remove_page_numbers` on the Python bindings — this issue is the second half of making furniture handling operable: choice there, visibility here.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in markdown/mod.rs at the existing furniture::strip_header_footer_lines gate, then inspect MarkdownConversionOutput and PdfProcessResult plus the Python, Node, and WASM result wrappers. Confirm the per-page versus total representation and whether folio removals share the count before tracing the result through each requested binding. Done means removed-line counts are visible in the agreed result surfaces without changing furniture.rs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, python, rust, wasm
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100