firecrawl / firecrawl/pdf-inspector

detect_columns misses sparse two-column pages: item-count floors hide large-print book spreads

Open
#462 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Summary

`detect_columns` never runs its geometry on pages with fewer than 20 text items, and every fallback path has an equal or higher floor — so sparse two-column pages (large-print children's books: one PDF page = two book pages, ~7 lines per column) are always reported single-column, and the extracted markdown interleaves the two columns line-by-line.

## Details

In `extractor/layout.rs` (v1.17.0):

- `detect_columns` early-returns a single region when `page_items.len() < 20` (line ~151).
- The relative-valley fallback needs `page_items.len() >= 30`.
- The XY-cut fallback (`try_xy_cut_split`) needs `page_items.len() >= 20`, plus `MIN_ITEMS_MAJOR = 10` on the major side.
- The prose gate (`columns_have_prose`) needs `MIN_LINES = 8` per column.

A large-print book spread has ~7 lines per column — it falls under *every* floor, no matter how obvious the gutter.

## Real-world example

A Catalan children's book PDF (5 pages, Mixed type). Page geometry from `extract_text_with_positions_mem` (text items only; one image placeholder per page filtered out):

```
page 3: 15 items
x= 65.2.. 271.8 y=492.7 "de mans senzills, com el de fer "
x= 65.2.. 280.8 y=470.7 "desaparéixer una moneda entre "
x= 83.6.. 266.7 y=514.7 "Va començar aprenent jocs "
x= 208.6.. 320.3 y=275.2 "...ara no la veieu!"
x= 416.7.. 573.8 y=492.7 "on ningú s'ho esperava:"
x= 435.1.. 636.7 y=514.7 "I, en acabant, fer-la aparéixer "
x= 435.1.. 648.0 y=470.7 "—Atenció! Què tens ací, darrere "
...
```

Left cluster x≈65–320, right cluster x≈415–710 — a **~95pt gutter**, line-scale items, 8–17 items per page. Trivially separable by a largest-gap sweep, but below every item-count floor. The two columns share y-coordinates, so single-column grouping merges them line-by-line into mid-word interleaved text ("Va començar aprenent jocs I, en acabant, fer-la aparéixer…").

## Suggestion

The floors exist to keep sparse single-column pages (covers, title pages) from false-splitting — that concern is real, but it could be preserved while admitting this class, e.g.:

- Let `try_xy_cut_split` run at lower item counts (it needs no histogram density) with scaled floors — e.g. `MIN_ITEMS_MAJOR/MINOR ≈ 3`, but require a *wider* gap for sparse pages (absolute ≥ ~24pt or ≥ ~4% of page width) so only unambiguous gutters qualify.
- Or gate a sparse-page path on the gap being large relative to both column widths, keeping dense-page behavior unchanged.

Happy to provide the sample PDF privately if useful for a regression test.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in extractor/layout.rs at detect_columns, then inspect try_xy_cut_split and the relative-valley fallback to understand how their item-count floors and gap checks interact. Reproduce the sparse two-column geometry described in the issue, and consider how to recognize its wide gutter without false-splitting covers or title pages; done means the columns extract in order while existing sparse single-column behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.