firecrawl / firecrawl/pdf-inspector

Sparse/grouped table rows collapse into a single Markdown row with concatenated cells

Open
#229 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Summary

On tables where rows have differing populated-cell patterns — a group-level row with most cells empty, then detail rows beneath it — Markdown extraction **collapses several physical rows into a single table row**, concatenating each column's values inside one cell.

A uniformly-populated table with the same columns extracts perfectly, so this is specific to the sparse shape rather than to the columns or the content.

## Reproduction

Fully self-contained (`pip install pdf-inspector reportlab`). It builds two PDFs with the same columns, differing only in shape — one sparse, one uniform:

```python
from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas

W, H = letter
X = [48, 108, 150, 215, 300, 362, 424, 470, 520, 556]
HEADERS = ["Order Date", "Suffix", "Line Section", "Item Code", "Description",
"Status", "Unit Cost", "Freight Charge", "Tax", "Total Billed"]
DETAIL = [
("WIDGET ASSEMBLY", "$482,110.40", "$0.00", "$9,215.75", "$491,326.15"),
("BRACKET SET", "$0.00", "$0.00", "$0.00", "$0.00"),
("CONTROL MODULE", "$31,905.22", "$4,100.00", "$0.00", "$36,005.22"),
]

# --- sparse: group row with empty cells, interleaved note, then detail rows ---
c = canvas.Canvas("sparse.pdf", pagesize=letter)
c.setFont("Helvetica", 7)
y = H - 90
c.drawString(X[1], y, "Account"); c.drawString(X[8], y, "Adjust") # two-line header
y -= 10
for xi, h in zip(X, HEADERS):
c.drawString(xi, y, h)
y -= 16
c.drawString(X[0], y, "03/14/2024") # group row: most cells empty
c.drawString(X[3], y, "IC-1048")
c.drawString(X[5], y, "Shipped")
y -= 12
c.drawString(X[4], y, "Shipment was delayed in transit and re-routed through the hub.")
y -= 12
for desc, unit, freight, tax, total in DETAIL:
c.drawString(X[3], y, "IC-1048"); c.drawString(X[4], y, desc); c.drawString(X[5], y, "Shipped")
c.drawRightString(X[7] - 6, y, unit); c.drawRightString(X[8] - 6, y, freight)
c.drawRightString(X[9] - 6, y, tax); c.drawRightString(X[9] + 34, y, total)
y -= 12
c.showPage(); c.save()

import pdf_inspector
print(pdf_inspector.extract_pages_markdown_bytes(open("sparse.pdf","rb").read()).pages[0].markdown)
```

### Actual output — all four rows collapse into one

```
|Order Date|Suffix|Item Code|Description|Status|Unit Cost|Freight Charge|Tax|Total Billed|
|---|---|---|---|---|---|---|---|---|
|03/14/2024||IC-1048 IC-1048 IC-1048 IC-1048|Shipment was delayed in transit and re-routed through the hub. WIDGET ASSEMBLY BRACKET SET CONTROL MODULE|Shipped Shipped Shipped Shipped|$482,110.40 $0.00 $31,905.22|$0.00 $0.00 $4,100.00|$9,215.75 $0.00 $0.00|$491,326.15 $0.00 $36,005.22|
```

Three distinct line items are now one row. `$482,110.40 $0.00 $31,905.22` is a single cell — three different orders' unit costs concatenated.

### Control — the same columns, uniformly populated, extract correctly

Building the same table with every row fully populated and no interleaved note gives exactly the right answer:

```
|Order Date|Item Code|Description|Status|Unit Cost|Qty|Tax|Ref|
|---|---|---|---|---|---|---|---|
|03/14/2024|IC-1048|WIDGET ASSEMBLY|Shipped|$482,110.40|$0.00|$9,215.75|N/A|
|05/02/2024|IC-1049|BRACKET SET|Shipped|$0.00|$0.00|$0.00|N/A|
|07/19/2024|IC-1050|CONTROL MODULE|Open|$31,905.22|$4,100.00|$0.00|N/A|
```

So the trigger is the sparse/grouped shape plus the interleaved free-text line, not the column layout.

On real documents the same bug appears in a milder form — adjacent numeric columns merging pairwise into one cell (e.g. `$0.00 $16,134.34`) rather than the whole table collapsing — but it's the same sparse-row shape driving it.

## Why it matters

The value of table extraction is knowing which column and which row each value belongs to. A collapsed cell forces the consumer to guess, and multiple readings are equally plausible.

The failure is silent and confident: the output is well-formed Markdown asserting a structure that doesn't match the document. Nothing raises, and row-count validation won't catch it, because the row count looks plausible — there just aren't enough of them.

Since the README benchmarks tables via TEDS, this shape may be under-represented in the current corpus. TEDS penalises a merged-but-present cell considerably less than the practical impact warrants, so a benchmark can look healthy while this case is broken.

## Suggested fix

Seed row boundaries from the text items' y-positions and column boundaries from their x-positions, rather than inferring either from rendered density — the geometry is already available and unambiguous here.

When a boundary is genuinely ambiguous, prefer splitting over merging: an extra empty row or column is trivially recoverable downstream, whereas a merged cell has lost information that can't be reconstructed.

## Note

For what it's worth, liteparse 2.10.1 exhibits a similar collapse on the same file, so this seems to be a genuinely hard shape rather than something the field has solved and this library hasn't.

## Environment

- `pdf-inspector==0.2.6` (PyPI)
- Python 3.13
- macOS 26.6, arm64

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the self-contained ReportLab reproduction and inspect the output from extract_pages_markdown_bytes. Trace how sparse text items are grouped into Markdown rows and columns, then verify that the group row, interleaved note, and three detail rows remain distinct without concatenated cells.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.