firecrawl / firecrawl/pdf-inspector

Distinct numeric value columns merged into one cell for one row per PDF (customs FDI statistical tables)

Open
#419 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

# Bug report: distinct numeric value columns merged in heuristic table extraction

## Summary

Two adjacent numeric value columns (`Reporting month.Value (USD)` and `Year to date.Volume`) are merged into a single cell for exactly one row per PDF, on Vietnamese Customs FDI statistical reports (`Table: 17B/TCHQ`). Reproduced on **pdf-inspector v1.15.0** (latest release, 2026-08-17) and on `master` (grid.rs unchanged since the merge pass was introduced).

## Environment

- pdf-inspector v1.15.0 (pip `pdf-inspector`), Python 3.13
- `extract_pages_markdown()` on a text-based 2-page PDF (78 KB, `classify_pdf → text_based, confidence 1.00`)

## Repro

Fixture: customs FDI preliminary report — July 2026 import, row 6 "Coal" (also reproduced on June 2026 import PDF; both attached). Both are real production PDFs from the Vietnam General Department of Customs.

```python
from pdf_inspector import extract_pages_markdown

r = extract_pages_markdown("july_import.pdf")
md = "\n".join(p.markdown for p in r.pages)
for line in md.splitlines():
if "|6 Coal|" in line:
print(line)
```

Actual (bug):

```
|6 Coal|Ton|1,709,851|246,515,548 11,734,697||1,555,338,055|
```

Expected: `rep_val` and `ytd_vol` as separate cells:

```
|6 Coal|Ton|1,709,851|246,515,548|11,734,697|1,555,338,055|
```

Every other row in the same table extracts correctly (34 rows total; only row 6 merges). Export-direction PDFs (different table layout) extract cleanly — 0 merged rows. The same Coal-row defect reproduces on June 2026 import: `|6 Coal|Ton|1,661,481|236,480,212 10,024,724||1,302,660,618|`.

## Root cause analysis

PDF word geometry (from `pdfplumber`, x0/x1 coordinates in pt) for the Coal row vs a clean row:

```
Wheats row value gaps: [22.5, 13.6, 20.6] min 13.6 pt
Coal row value gaps: [17.0, 8.1, 12.3] min 8.1 pt <-- abnormal
```

The Coal row's `rep_val → ytd_vol` gap is **8.1 pt** (every other row ≥ 12.3 pt). This is deterministic in the source PDF (same 8.1 pt signature in both June and July; the Coal commodity name is short, shifting the value-column alignment).

In `src/tables/grid.rs`, `merge_numeric_adjacent_clusters`:

```rust
let merge_dist = threshold * 1.5; // default threshold 25pt -> 37.5pt
// merge if dense.numeric_frac > 0.50 && sparse.count <= dense.count / 2 && sparse.count <= 5
```

Both adjacent clusters are predominantly numeric; the cluster centers land within `merge_dist` (37.5 pt) because the Coal row's 8.1 pt gap pulls the centers together, so the merge pass collapses the two value columns for this row. For all other rows the gap (≥12.3 pt) keeps the centers far enough apart to avoid the merge — but only just (12.3 pt is still well under 37.5 pt; the algorithm happens to survive on center distribution, not on a hard gap check).

## Suggested fix

The numeric merge pass exists to fix header-vs-data splits (`"header text"` next to a dense numeric column). Merging **two predominantly numeric clusters** into one is never the intended behavior. Propose guarding `merge_numeric_adjacent_clusters` so it does not merge when **both** clusters are predominantly numeric:

```rust
let should_merge =
dense.numeric_frac > 0.50
&& sparse.count <= dense.count / 2
&& sparse.count <= 5
&& !(info_a.numeric_frac > 0.50 && info_b.numeric_frac > 0.50); // new guard
```

This preserves the header-fix behavior while preventing numeric-column collapse. Alternative (weaker): require `dist > some_min_gap` when both numeric.

## Fixtures

- `july_import.pdf` (78,419 bytes) — July 2026, import, reproduces
- `2026_06_import_preliminary.pdf` (78,347 bytes) — June 2026, import, reproduces
- Control (clean): `2026_06_export_preliminary.pdf` — export direction, no merge

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/tables/grid.rs at merge_numeric_adjacent_clusters and reproduce the issue with july_import.pdf or 2026_06_import_preliminary.pdf through extract_pages_markdown(). Check the merge decision for the Coal row against a clean row and the export control PDF. Done means the two numeric columns remain separate while the existing header-vs-data merge behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.