firecrawl / firecrawl/pdf-inspector
detect_columns misses sparse two-column pages: item-count floors hide large-print book spreads
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Summary
`detect_columns` never runs its geometry on pages with fewer than 20 text items, and every fallback path has an equal or higher floor — so sparse two-column pages (large-print children's books: one PDF page = two book pages, ~7 lines per column) are always reported single-column, and the extracted markdown interleaves the two columns line-by-line.
## Details
In `extractor/layout.rs` (v1.17.0):
- `detect_columns` early-returns a single region when `page_items.len() < 20` (line ~151).
- The relative-valley fallback needs `page_items.len() >= 30`.
- The XY-cut fallback (`try_xy_cut_split`) needs `page_items.len() >= 20`, plus `MIN_ITEMS_MAJOR = 10` on the major side.
- The prose gate (`columns_have_prose`) needs `MIN_LINES = 8` per column.
A large-print book spread has ~7 lines per column — it falls under *every* floor, no matter how obvious the gutter.
## Real-world example
A Catalan children's book PDF (5 pages, Mixed type). Page geometry from `extract_text_with_positions_mem` (text items only; one image placeholder per page filtered out):
```
page 3: 15 items
x= 65.2.. 271.8 y=492.7 "de mans senzills, com el de fer "
x= 65.2.. 280.8 y=470.7 "desaparéixer una moneda entre "
x= 83.6.. 266.7 y=514.7 "Va començar aprenent jocs "
x= 208.6.. 320.3 y=275.2 "...ara no la veieu!"
x= 416.7.. 573.8 y=492.7 "on ningú s'ho esperava:"
x= 435.1.. 636.7 y=514.7 "I, en acabant, fer-la aparéixer "
x= 435.1.. 648.0 y=470.7 "—Atenció! Què tens ací, darrere "
...
```
Left cluster x≈65–320, right cluster x≈415–710 — a **~95pt gutter**, line-scale items, 8–17 items per page. Trivially separable by a largest-gap sweep, but below every item-count floor. The two columns share y-coordinates, so single-column grouping merges them line-by-line into mid-word interleaved text ("Va començar aprenent jocs I, en acabant, fer-la aparéixer…").
## Suggestion
The floors exist to keep sparse single-column pages (covers, title pages) from false-splitting — that concern is real, but it could be preserved while admitting this class, e.g.:
- Let `try_xy_cut_split` run at lower item counts (it needs no histogram density) with scaled floors — e.g. `MIN_ITEMS_MAJOR/MINOR ≈ 3`, but require a *wider* gap for sparse pages (absolute ≥ ~24pt or ≥ ~4% of page width) so only unambiguous gutters qualify.
- Or gate a sparse-page path on the gap being large relative to both column widths, keeping dense-page behavior unchanged.
Happy to provide the sample PDF privately if useful for a regression test.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in extractor/layout.rs at detect_columns, then inspect try_xy_cut_split and the relative-valley fallback to understand how their item-count floors and gap checks interact. Reproduce the sparse two-column geometry described in the issue, and consider how to recognize its wide gutter without false-splitting covers or title pages; done means the columns extract in order while existing sparse single-column behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100