firecrawl / firecrawl/pdf-inspector
Table row is dropped into prose when its label starts with a footnote marker (is_footnote_row only checks the text before the first ")")
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Summary
A data row is removed from a detected table and emitted as a loose footnote line whenever its first cell *begins* with a numeric footnote marker (`1) Heat output`). The values survive as text, so nothing errors and text-recall checks pass, but they lose their columns.
On a corpus of 30 technical PDFs this drops **52 of 465** data rows (11%).
## Reproduction
Deterministic, no sample file needed. `make_repro.py`:
```python
from reportlab.lib.pagesizes import A4
from reportlab.lib.units import mm
from reportlab.platypus import SimpleDocTemplate, Table, TableStyle
from reportlab.lib import colors
HEADER = ["Measurement", "Unit", "Q1", "Q2", "Q3"]
ROWS = [
["Mass flow", "kg/s", "3,10E+00", "3,35E-01", "4,23E-01"],
["1) Heat output", "kW", "2,41E+02", "2,74E+01", "6,69E+00"],
["Pressure drop", "kPa", "1,91E+03", "3,96E+02", "8,67E+01"],
]
doc = SimpleDocTemplate("repro_footnote_marker.pdf", pagesize=A4,
leftMargin=15*mm, rightMargin=15*mm,
topMargin=15*mm, bottomMargin=15*mm)
t = Table([HEADER] + ROWS, colWidths=[40*mm, 20*mm, 30*mm, 30*mm, 30*mm])
t.setStyle(TableStyle([
("GRID", (0, 0), (-1, -1), 0.5, colors.black),
("FONTNAME", (0, 0), (-1, -1), "Helvetica"),
("FONTSIZE", (0, 0), (-1, -1), 8),
]))
doc.build([t])
```
```
python make_repro.py
cargo run --release --bin pdf2md -- repro_footnote_marker.pdf /dev/stdout
```
**Expected**
```
|Measurement|Unit|Q1|Q2|Q3|
|---|---|---|---|---|
|Mass flow|kg/s|3,10E+00|3,35E-01|4,23E-01|
|1) Heat output|kW|2,41E+02|2,74E+01|6,69E+00|
|Pressure drop|kPa|1,91E+03|3,96E+02|8,67E+01|
```
**Actual**
```
|Measurement|Unit|Q1|Q2|Q3|
|---|---|---|---|---|
|Mass flow|kg/s|3,10E+00|3,35E-01|4,23E-01|
|Pressure drop|kPa|1,91E+03|3,96E+02|8,67E+01|
1) Heat output kW 2,41E+02 2,74E+01 6,69E+00
```
## Cause
`clean_table_cells` (`src/tables/format.rs:270`) classifies a row from its first cell alone:
```rust
let first_cell = row.first().map(|s| s.trim()).unwrap_or("");
if is_footnote_row(first_cell) {
...
continue;
}
```
`is_footnote_row` (`src/tables/format.rs:442`) inspects only the text *before* the first `)`:
```rust
if let Some(paren_idx) = trimmed.find(')') {
let num_part = &trimmed[..paren_idx];
if !num_part.is_empty() && num_part.chars().all(|c| c.is_ascii_digit()) {
return true; // is_footnote_row("1) Heat output") == true
}
}
```
There is no check that the rest of the row is data. In the corpus, the deleted rows carried 18 numeric cells each.
## Suggested fix
Guard the footnote branch so a row carrying numeric cells is never classified as a footnote:
```rust
let data_cells = row.iter().filter(|c| is_data_cell(c)).count();
if data_cells < MIN_DATA_CELLS_FOR_ROW && is_footnote_row(first_cell) {
```
`is_data_cell` needs to be a new helper — `financial::is_numeric_token` rejects scientific notation (`2,41E+02`). A threshold of 2 keeps borderline rows in the table.
| build | rows intact |
|---|---|
| `585d36e` | 400/465 (86.0%) |
| with the guard | 452/465 (97.2%) |
`cargo test` (full suite, unit + 152 integration over `tests/fixtures/`) goes 975 → 977 with two regression tests added; no existing test changes. `test_clean_table_cells_footnote_extracted` still passes. Can open a PR if useful.
## Environment
- pdf-inspector `585d36e`, version 0.1.7
- rustc 1.92.0 (ded5c06cf 2025-12-08)
- Linux 7.0.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/tables/format.rs at clean_table_cells around line 270 and is_footnote_row around line 442, then run make_repro.py and the shown cargo run command to reproduce the dropped row. Add regression coverage for numeric cells in footnote-marked rows and run cargo test; done means the 1) Heat output row remains in the table while test_clean_table_cells_footnote_extracted still passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100