firecrawl / firecrawl/pdf-inspector
Distinct numeric value columns merged into one cell for one row per PDF (customs FDI statistical tables)
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
# Bug report: distinct numeric value columns merged in heuristic table extraction
## Summary
Two adjacent numeric value columns (`Reporting month.Value (USD)` and `Year to date.Volume`) are merged into a single cell for exactly one row per PDF, on Vietnamese Customs FDI statistical reports (`Table: 17B/TCHQ`). Reproduced on **pdf-inspector v1.15.0** (latest release, 2026-08-17) and on `master` (grid.rs unchanged since the merge pass was introduced).
## Environment
- pdf-inspector v1.15.0 (pip `pdf-inspector`), Python 3.13
- `extract_pages_markdown()` on a text-based 2-page PDF (78 KB, `classify_pdf → text_based, confidence 1.00`)
## Repro
Fixture: customs FDI preliminary report — July 2026 import, row 6 "Coal" (also reproduced on June 2026 import PDF; both attached). Both are real production PDFs from the Vietnam General Department of Customs.
```python
from pdf_inspector import extract_pages_markdown
r = extract_pages_markdown("july_import.pdf")
md = "\n".join(p.markdown for p in r.pages)
for line in md.splitlines():
if "|6 Coal|" in line:
print(line)
```
Actual (bug):
```
|6 Coal|Ton|1,709,851|246,515,548 11,734,697||1,555,338,055|
```
Expected: `rep_val` and `ytd_vol` as separate cells:
```
|6 Coal|Ton|1,709,851|246,515,548|11,734,697|1,555,338,055|
```
Every other row in the same table extracts correctly (34 rows total; only row 6 merges). Export-direction PDFs (different table layout) extract cleanly — 0 merged rows. The same Coal-row defect reproduces on June 2026 import: `|6 Coal|Ton|1,661,481|236,480,212 10,024,724||1,302,660,618|`.
## Root cause analysis
PDF word geometry (from `pdfplumber`, x0/x1 coordinates in pt) for the Coal row vs a clean row:
```
Wheats row value gaps: [22.5, 13.6, 20.6] min 13.6 pt
Coal row value gaps: [17.0, 8.1, 12.3] min 8.1 pt <-- abnormal
```
The Coal row's `rep_val → ytd_vol` gap is **8.1 pt** (every other row ≥ 12.3 pt). This is deterministic in the source PDF (same 8.1 pt signature in both June and July; the Coal commodity name is short, shifting the value-column alignment).
In `src/tables/grid.rs`, `merge_numeric_adjacent_clusters`:
```rust
let merge_dist = threshold * 1.5; // default threshold 25pt -> 37.5pt
// merge if dense.numeric_frac > 0.50 && sparse.count <= dense.count / 2 && sparse.count <= 5
```
Both adjacent clusters are predominantly numeric; the cluster centers land within `merge_dist` (37.5 pt) because the Coal row's 8.1 pt gap pulls the centers together, so the merge pass collapses the two value columns for this row. For all other rows the gap (≥12.3 pt) keeps the centers far enough apart to avoid the merge — but only just (12.3 pt is still well under 37.5 pt; the algorithm happens to survive on center distribution, not on a hard gap check).
## Suggested fix
The numeric merge pass exists to fix header-vs-data splits (`"header text"` next to a dense numeric column). Merging **two predominantly numeric clusters** into one is never the intended behavior. Propose guarding `merge_numeric_adjacent_clusters` so it does not merge when **both** clusters are predominantly numeric:
```rust
let should_merge =
dense.numeric_frac > 0.50
&& sparse.count <= dense.count / 2
&& sparse.count <= 5
&& !(info_a.numeric_frac > 0.50 && info_b.numeric_frac > 0.50); // new guard
```
This preserves the header-fix behavior while preventing numeric-column collapse. Alternative (weaker): require `dist > some_min_gap` when both numeric.
## Fixtures
- `july_import.pdf` (78,419 bytes) — July 2026, import, reproduces
- `2026_06_import_preliminary.pdf` (78,347 bytes) — June 2026, import, reproduces
- Control (clean): `2026_06_export_preliminary.pdf` — export direction, no merge
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/tables/grid.rs at merge_numeric_adjacent_clusters and reproduce the issue with july_import.pdf or 2026_06_import_preliminary.pdf through extract_pages_markdown(). Check the merge decision for the Coal row against a clean row and the export control PDF. Done means the two numeric columns remain separate while the existing header-vs-data merge behavior is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100