firecrawl / firecrawl/pdf-inspector

extract_pages_markdown emits <u> tags that split a word when underline flag differs across positioned items in the same visual word

Open
#397 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

## Summary

A display heading laid out as several positioned text items can carry the `is_underline` flag on only one of those items. `extract_pages_markdown` (and `process_pdf`) then wraps just that item in `...`, splitting a single visual word with a tag boundary:

```
The Institutional Wealth Landscape
```

Rendered, `` never opens or closes on a non-word boundary, so this isn't formatting that was in the source — it's an artifact of how the layout engine grouped (or failed to group) items into a word before checking the underline flag.

## Environment

- pdf-inspector (Python binding) via `pip install pdf-inspector`
- Reproduced via `pdf_inspector.process_pdf()` / `extract_pages_markdown()`

## Reproduction

On an 11-page report exported from Chrome ("Save as PDF", producer `Skia/PDF`), page 1's title is `The Institutional Wealth Landscape`, styled as a single 42pt heading.

```python
import pdf_inspector
result = pdf_inspector.process_pdf("report.pdf")
print(result.markdown)
# ...
# # The Institutional Wealth Landscape
```

Checking the underlying positioned items confirms the split happens at the item level:

```python
items = pdf_inspector.extract_text_with_positions("report.pdf")
for t in items:
if t.text in ("The Institutional", "We", "alth Landscape"):
print(t.text, t.is_underline, t.x, t.font_size)
# 'The Institutional' is_underline=False x=45.0 size=42.0
# 'We' is_underline=True x=45.0 size=42.0
# 'alth Landscape' is_underline=False x=101.8 size=42.0
```

Three items, contiguous, same font size, same baseline — clearly one visual word ("Wealth") split across two content-stream text-showing operators, with `is_underline` true on only the first fragment. Six other headings in the same document exhibit the same pattern (`01 Geopolitical and macro risk`, `Master source index`, etc.) — those happen to be whole-phrase, so they render fine, but the underlying cause is the same per-item flag not being reconciled before Markdown conversion.

## Expected behavior

Either:
1. The word-grouping step that assembles positioned items into `TextLine`/word units should also reconcile `is_underline` across items that form one word (e.g. before word-boundary characters, or via majority/first-non-trivial fragment), so the tag wraps the whole word or none of it, or
2. If the items are being treated as separate underline runs on purpose, the Markdown converter should not open/close a `` tag on a non-word-boundary character.

## Impact

Cosmetic on its own, but it corrupts every downstream consumer that treats the Markdown as ground truth text — search indexing, embeddings, and diffing all see `Wealth` broken into two tokens (`We`, `alth`) with an HTML tag between them. In our case it also required a post-processing repair step before the Markdown could be treated as reliable extraction output.

## Suggested severity

Low-to-medium: doesn't affect classification or table extraction, but it's silent — nothing in the output flags the corruption, so it can only be caught by comparing against the source.

---

Happy to share the source PDF (Skia/PDF producer) privately if it helps reproduce — it isn't public but I can send a minimal repro extracted from it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing pdf_inspector.process_pdf() and extract_pages_markdown() into the positioned-item grouping and Markdown conversion described in the report. Use the reported report.pdf pattern or a minimal reproduction to inspect extract_text_with_positions() output. Done means a visual word such as “Wealth” is not split by an underline tag in the extracted Markdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.