firecrawl / firecrawl/pdf-inspector

Feature request: report removed header/footer lines (audit trail for furniture stripping)

Open
#459 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Problem

`strip_headers_footers` (and the folio filtering) remove text with no trace of what or how much was removed. For ingestion pipelines that persist extracted text (RAG, archival), silent removal is hard to operate: when a customer asks "where did my footer text go" — or worse, when the furniture heuristic misfires on a boundary case — there is nothing to audit.

Our pipeline stamps every page that lost lines with a removed-count in page metadata, so removals are visible without diffing outputs. We currently do this with a layout-model-based extractor and would like parity when processing via pdf-inspector.

## Proposal

Add a per-page removed-line count to `PdfProcessResult`, e.g.:

```rust
/// 1-indexed page → number of text lines removed as running headers/footers.
pub removed_header_footer_lines: Vec<(u32, u32)>,
```

Sketch (happy to implement if the direction sounds right):

- Compute per-page counts at the existing gate in `markdown/mod.rs` (diff the per-page line counts around `furniture::strip_header_footer_lines`) — no change to `furniture.rs` itself.
- Thread them through `MarkdownConversionOutput` (the struct already exists for `markdown` + `detected_tables`).
- Surface on `PdfProcessResult` and the Python/Node/WASM result wrappers (or Python-only to start, if you prefer keeping the other bindings lean).

Open questions where I'd take your steer:

1. Total vs per-page (per-page is what an audit trail needs; total is cheaper to plumb).
2. Whether folio removals (`select_items_with_document_folio_context`) should be counted in the same field or reported separately — they are removed unconditionally, so an audit entry is arguably even more valuable there.
3. Whether `MarkdownConversionOutput` is the right vehicle or you'd rather grow the `MarkdownDocumentContext` with an out-slot.

Related: the sibling PR exposing `strip_headers_footers` / `remove_page_numbers` on the Python bindings — this issue is the second half of making furniture handling operable: choice there, visibility here.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in markdown/mod.rs at the existing furniture::strip_header_footer_lines gate, then inspect MarkdownConversionOutput and PdfProcessResult plus the Python, Node, and WASM result wrappers. Confirm the per-page versus total representation and whether folio removals share the count before tracing the result through each requested binding. Done means removed-line counts are visible in the agreed result surfaces without changing furniture.rs.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, python, rust, wasm
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.