firecrawl / firecrawl/pdf-inspector

Table row is dropped into prose when its label starts with a footnote marker (is_footnote_row only checks the text before the first ")")

Open
#288 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Summary

A data row is removed from a detected table and emitted as a loose footnote line whenever its first cell *begins* with a numeric footnote marker (`1) Heat output`). The values survive as text, so nothing errors and text-recall checks pass, but they lose their columns.

On a corpus of 30 technical PDFs this drops **52 of 465** data rows (11%).

## Reproduction

Deterministic, no sample file needed. `make_repro.py`:

```python
from reportlab.lib.pagesizes import A4
from reportlab.lib.units import mm
from reportlab.platypus import SimpleDocTemplate, Table, TableStyle
from reportlab.lib import colors

HEADER = ["Measurement", "Unit", "Q1", "Q2", "Q3"]
ROWS = [
["Mass flow", "kg/s", "3,10E+00", "3,35E-01", "4,23E-01"],
["1) Heat output", "kW", "2,41E+02", "2,74E+01", "6,69E+00"],
["Pressure drop", "kPa", "1,91E+03", "3,96E+02", "8,67E+01"],
]

doc = SimpleDocTemplate("repro_footnote_marker.pdf", pagesize=A4,
leftMargin=15*mm, rightMargin=15*mm,
topMargin=15*mm, bottomMargin=15*mm)
t = Table([HEADER] + ROWS, colWidths=[40*mm, 20*mm, 30*mm, 30*mm, 30*mm])
t.setStyle(TableStyle([
("GRID", (0, 0), (-1, -1), 0.5, colors.black),
("FONTNAME", (0, 0), (-1, -1), "Helvetica"),
("FONTSIZE", (0, 0), (-1, -1), 8),
]))
doc.build([t])
```

```
python make_repro.py
cargo run --release --bin pdf2md -- repro_footnote_marker.pdf /dev/stdout
```

**Expected**

```
|Measurement|Unit|Q1|Q2|Q3|
|---|---|---|---|---|
|Mass flow|kg/s|3,10E+00|3,35E-01|4,23E-01|
|1) Heat output|kW|2,41E+02|2,74E+01|6,69E+00|
|Pressure drop|kPa|1,91E+03|3,96E+02|8,67E+01|
```

**Actual**

```
|Measurement|Unit|Q1|Q2|Q3|
|---|---|---|---|---|
|Mass flow|kg/s|3,10E+00|3,35E-01|4,23E-01|
|Pressure drop|kPa|1,91E+03|3,96E+02|8,67E+01|

1) Heat output kW 2,41E+02 2,74E+01 6,69E+00
```

## Cause

`clean_table_cells` (`src/tables/format.rs:270`) classifies a row from its first cell alone:

```rust
let first_cell = row.first().map(|s| s.trim()).unwrap_or("");
if is_footnote_row(first_cell) {
...
continue;
}
```

`is_footnote_row` (`src/tables/format.rs:442`) inspects only the text *before* the first `)`:

```rust
if let Some(paren_idx) = trimmed.find(')') {
let num_part = &trimmed[..paren_idx];
if !num_part.is_empty() && num_part.chars().all(|c| c.is_ascii_digit()) {
return true; // is_footnote_row("1) Heat output") == true
}
}
```

There is no check that the rest of the row is data. In the corpus, the deleted rows carried 18 numeric cells each.

## Suggested fix

Guard the footnote branch so a row carrying numeric cells is never classified as a footnote:

```rust
let data_cells = row.iter().filter(|c| is_data_cell(c)).count();
if data_cells < MIN_DATA_CELLS_FOR_ROW && is_footnote_row(first_cell) {
```

`is_data_cell` needs to be a new helper — `financial::is_numeric_token` rejects scientific notation (`2,41E+02`). A threshold of 2 keeps borderline rows in the table.

| build | rows intact |
|---|---|
| `585d36e` | 400/465 (86.0%) |
| with the guard | 452/465 (97.2%) |

`cargo test` (full suite, unit + 152 integration over `tests/fixtures/`) goes 975 → 977 with two regression tests added; no existing test changes. `test_clean_table_cells_footnote_extracted` still passes. Can open a PR if useful.

## Environment

- pdf-inspector `585d36e`, version 0.1.7
- rustc 1.92.0 (ded5c06cf 2025-12-08)
- Linux 7.0.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/tables/format.rs at clean_table_cells around line 270 and is_footnote_row around line 442, then run make_repro.py and the shown cargo run command to reproduce the dropped row. Add regression coverage for numeric cells in footnote-marked rows and run cargo test; done means the 1) Heat output row remains in the table while test_clean_table_cells_footnote_extracted still passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.