firecrawl / firecrawl/pdf-inspector

Markdown list detection only recognizes 6 bullet glyph types; extractor modules already handle 10+, so ▪/‣/◆ list items are silently demoted to plain paragraphs

Open
#475 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
19.2k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

Summary

is_list_item() and format_list_item() in src/markdown/classify.rs only recognize a small set of bullet glyphs (• ○ ● ◦ plus - *), while other modules in the same crate recognize a much wider set. A list item like ▪ Item is therefore not detected as a list at all and is emitted as a plain paragraph.

Bullet glyph sets across modules (as of current main)

Module Glyphs Purpose
src/markdown/classify.rs:73-93 (is_list_item, starts_with_bullet_marker) • ○ ● ◦ - * Decides whether a line is a list item in markdown output
src/markdown/classify.rs:130 (format_list_item) • ○ ● ◦ Converts the marker to a markdown -
src/extractor/layout.rs:798 (is_list_marker_column) • ● ○ ◦ ▪ ▫ ◆ ◇ ■ □ Rejects spurious column-break candidates on bullet columns
src/extractor/underline.rs:400 (is_bare_list_marker) • ◦ ▪ ▫ ‣ ⁃ ● ○ ■ □ - * Excludes bare markers from underline/strikethrough detection
src/extractor/mod.rs:637 (is_standalone_bullet_text) • ○ ● ◦ Merges standalone bullet glyphs with following text
src/tables/mod.rs:1319 • ● · Numeric-cell heuristics for table detection
src/vision/fusion.rs:763 • ● ○ ◦ ▪ - — Vision fusion bullet set
README "Markdown output" table • - * ○ ● ◦ Documented behavior

The extractor knows that ▪ ▫ ◆ ◇ ■ □ ‣ ⁃ · are list markers (it refuses to split bullet columns on them, and excludes them from underline detection), but the markdown layer never receives that knowledge.

Reproduction

Verified on current main (added a scratch unit test, since removed):

is_list_item("▪ Item")                // false  ← expected: true
starts_with_bullet_marker("▪ Item")    // false  ← expected: true
format_list_item("▪ Item")             // "▪ Item"  ← expected: "- Item"
is_list_item("‣ Item")                 // false  ← expected: true
is_list_item("◆ Item")                 // false  ← expected: true

So a PDF list rendered with markers produces:

▪ First item
▪ Second item
▪ Third item

instead of:

- First item
- Second item
- Third item

Downstream consequence: since is_list_item() gates heading classification (src/markdown/heading.rs:291) and code classification, a short bold -bullet line may additionally be misclassified as a heading.

Proposed direction

Add a single shared bullet-marker constant (e.g. pub(crate) const BULLET_MARKERS: &[char] in src/markdown/classify.rs or a common module) and use it in is_list_item(), starts_with_bullet_marker(), format_list_item(), is_list_marker_column(), is_bare_list_marker(), and is_standalone_bullet_text(), so glyph sets can no longer drift apart between modules. The extractor sets suggest the full working set is at least:

• ● ○ ◦ ▪ ▫ ◆ ◇ ■ □ ‣ ⁃ · - *

· (U+00B7 MIDDLE DOT) is intentionally ambiguous (it also appears in normal prose), so it may warrant separate treatment rather than inclusion in the automatic list-marker set.

Environment

  • pdf-inspector main (checked out today)
  • Only library-level unit tests; no specific PDF needed to reproduce (works from pure string inputs)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/markdown/classify.rs, especially is_list_item(), starts_with_bullet_marker(), and format_list_item(), then compare the marker checks in src/extractor/layout.rs, src/extractor/underline.rs, and src/extractor/mod.rs. Add coverage for ▪, ‣, and ◆ using the library-level unit-test setup described in the issue, and verify those markers become Markdown list items without changing ambiguous · handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown, rust
Domain
backend, content
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.