firecrawl / firecrawl/pdf-inspector
Sparse/grouped table rows collapse into a single Markdown row with concatenated cells
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Summary
On tables where rows have differing populated-cell patterns — a group-level row with most cells empty, then detail rows beneath it — Markdown extraction **collapses several physical rows into a single table row**, concatenating each column's values inside one cell.
A uniformly-populated table with the same columns extracts perfectly, so this is specific to the sparse shape rather than to the columns or the content.
## Reproduction
Fully self-contained (`pip install pdf-inspector reportlab`). It builds two PDFs with the same columns, differing only in shape — one sparse, one uniform:
```python
from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas
W, H = letter
X = [48, 108, 150, 215, 300, 362, 424, 470, 520, 556]
HEADERS = ["Order Date", "Suffix", "Line Section", "Item Code", "Description",
"Status", "Unit Cost", "Freight Charge", "Tax", "Total Billed"]
DETAIL = [
("WIDGET ASSEMBLY", "$482,110.40", "$0.00", "$9,215.75", "$491,326.15"),
("BRACKET SET", "$0.00", "$0.00", "$0.00", "$0.00"),
("CONTROL MODULE", "$31,905.22", "$4,100.00", "$0.00", "$36,005.22"),
]
# --- sparse: group row with empty cells, interleaved note, then detail rows ---
c = canvas.Canvas("sparse.pdf", pagesize=letter)
c.setFont("Helvetica", 7)
y = H - 90
c.drawString(X[1], y, "Account"); c.drawString(X[8], y, "Adjust") # two-line header
y -= 10
for xi, h in zip(X, HEADERS):
c.drawString(xi, y, h)
y -= 16
c.drawString(X[0], y, "03/14/2024") # group row: most cells empty
c.drawString(X[3], y, "IC-1048")
c.drawString(X[5], y, "Shipped")
y -= 12
c.drawString(X[4], y, "Shipment was delayed in transit and re-routed through the hub.")
y -= 12
for desc, unit, freight, tax, total in DETAIL:
c.drawString(X[3], y, "IC-1048"); c.drawString(X[4], y, desc); c.drawString(X[5], y, "Shipped")
c.drawRightString(X[7] - 6, y, unit); c.drawRightString(X[8] - 6, y, freight)
c.drawRightString(X[9] - 6, y, tax); c.drawRightString(X[9] + 34, y, total)
y -= 12
c.showPage(); c.save()
import pdf_inspector
print(pdf_inspector.extract_pages_markdown_bytes(open("sparse.pdf","rb").read()).pages[0].markdown)
```
### Actual output — all four rows collapse into one
```
|Order Date|Suffix|Item Code|Description|Status|Unit Cost|Freight Charge|Tax|Total Billed|
|---|---|---|---|---|---|---|---|---|
|03/14/2024||IC-1048 IC-1048 IC-1048 IC-1048|Shipment was delayed in transit and re-routed through the hub. WIDGET ASSEMBLY BRACKET SET CONTROL MODULE|Shipped Shipped Shipped Shipped|$482,110.40 $0.00 $31,905.22|$0.00 $0.00 $4,100.00|$9,215.75 $0.00 $0.00|$491,326.15 $0.00 $36,005.22|
```
Three distinct line items are now one row. `$482,110.40 $0.00 $31,905.22` is a single cell — three different orders' unit costs concatenated.
### Control — the same columns, uniformly populated, extract correctly
Building the same table with every row fully populated and no interleaved note gives exactly the right answer:
```
|Order Date|Item Code|Description|Status|Unit Cost|Qty|Tax|Ref|
|---|---|---|---|---|---|---|---|
|03/14/2024|IC-1048|WIDGET ASSEMBLY|Shipped|$482,110.40|$0.00|$9,215.75|N/A|
|05/02/2024|IC-1049|BRACKET SET|Shipped|$0.00|$0.00|$0.00|N/A|
|07/19/2024|IC-1050|CONTROL MODULE|Open|$31,905.22|$4,100.00|$0.00|N/A|
```
So the trigger is the sparse/grouped shape plus the interleaved free-text line, not the column layout.
On real documents the same bug appears in a milder form — adjacent numeric columns merging pairwise into one cell (e.g. `$0.00 $16,134.34`) rather than the whole table collapsing — but it's the same sparse-row shape driving it.
## Why it matters
The value of table extraction is knowing which column and which row each value belongs to. A collapsed cell forces the consumer to guess, and multiple readings are equally plausible.
The failure is silent and confident: the output is well-formed Markdown asserting a structure that doesn't match the document. Nothing raises, and row-count validation won't catch it, because the row count looks plausible — there just aren't enough of them.
Since the README benchmarks tables via TEDS, this shape may be under-represented in the current corpus. TEDS penalises a merged-but-present cell considerably less than the practical impact warrants, so a benchmark can look healthy while this case is broken.
## Suggested fix
Seed row boundaries from the text items' y-positions and column boundaries from their x-positions, rather than inferring either from rendered density — the geometry is already available and unambiguous here.
When a boundary is genuinely ambiguous, prefer splitting over merging: an extra empty row or column is trivially recoverable downstream, whereas a merged cell has lost information that can't be reconstructed.
## Note
For what it's worth, liteparse 2.10.1 exhibits a similar collapse on the same file, so this seems to be a genuinely hard shape rather than something the field has solved and this library hasn't.
## Environment
- `pdf-inspector==0.2.6` (PyPI)
- Python 3.13
- macOS 26.6, arm64
Contributor guide
No contributing guide indexed for this repository
Research direction
Run the self-contained ReportLab reproduction and inspect the output from extract_pages_markdown_bytes. Trace how sparse text items are grouped into Markdown rows and columns, then verify that the group row, interleaved note, and three detail rows remain distinct without concatenated cells.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100