firecrawl / firecrawl/pdf-inspector
extract_pages_markdown emits <u> tags that split a word when underline flag differs across positioned items in the same visual word
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
## Summary
A display heading laid out as several positioned text items can carry the `is_underline` flag on only one of those items. `extract_pages_markdown` (and `process_pdf`) then wraps just that item in `...`, splitting a single visual word with a tag boundary:
```
The Institutional Wealth Landscape
```
Rendered, `` never opens or closes on a non-word boundary, so this isn't formatting that was in the source — it's an artifact of how the layout engine grouped (or failed to group) items into a word before checking the underline flag.
## Environment
- pdf-inspector (Python binding) via `pip install pdf-inspector`
- Reproduced via `pdf_inspector.process_pdf()` / `extract_pages_markdown()`
## Reproduction
On an 11-page report exported from Chrome ("Save as PDF", producer `Skia/PDF`), page 1's title is `The Institutional Wealth Landscape`, styled as a single 42pt heading.
```python
import pdf_inspector
result = pdf_inspector.process_pdf("report.pdf")
print(result.markdown)
# ...
# # The Institutional Wealth Landscape
```
Checking the underlying positioned items confirms the split happens at the item level:
```python
items = pdf_inspector.extract_text_with_positions("report.pdf")
for t in items:
if t.text in ("The Institutional", "We", "alth Landscape"):
print(t.text, t.is_underline, t.x, t.font_size)
# 'The Institutional' is_underline=False x=45.0 size=42.0
# 'We' is_underline=True x=45.0 size=42.0
# 'alth Landscape' is_underline=False x=101.8 size=42.0
```
Three items, contiguous, same font size, same baseline — clearly one visual word ("Wealth") split across two content-stream text-showing operators, with `is_underline` true on only the first fragment. Six other headings in the same document exhibit the same pattern (`01 Geopolitical and macro risk`, `Master source index`, etc.) — those happen to be whole-phrase, so they render fine, but the underlying cause is the same per-item flag not being reconciled before Markdown conversion.
## Expected behavior
Either:
1. The word-grouping step that assembles positioned items into `TextLine`/word units should also reconcile `is_underline` across items that form one word (e.g. before word-boundary characters, or via majority/first-non-trivial fragment), so the tag wraps the whole word or none of it, or
2. If the items are being treated as separate underline runs on purpose, the Markdown converter should not open/close a `` tag on a non-word-boundary character.
## Impact
Cosmetic on its own, but it corrupts every downstream consumer that treats the Markdown as ground truth text — search indexing, embeddings, and diffing all see `Wealth` broken into two tokens (`We`, `alth`) with an HTML tag between them. In our case it also required a post-processing repair step before the Markdown could be treated as reliable extraction output.
## Suggested severity
Low-to-medium: doesn't affect classification or table extraction, but it's silent — nothing in the output flags the corruption, so it can only be caught by comparing against the source.
---
Happy to share the source PDF (Skia/PDF producer) privately if it helps reproduce — it isn't public but I can send a minimal repro extracted from it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing pdf_inspector.process_pdf() and extract_pages_markdown() into the positioned-item grouping and Markdown conversion described in the report. Use the reported report.pdf pattern or a minimal reproduction to inspect extract_text_with_positions() output. Done means a visual word such as “Wealth” is not split by an underline tag in the extracted Markdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100