firecrawl / firecrawl/pdf-inspector
Markdown list detection only recognizes 6 bullet glyph types; extractor modules already handle 10+, so ▪/‣/◆ list items are silently demoted to plain paragraphs
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19.2k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
Summary
is_list_item() and format_list_item() in src/markdown/classify.rs only recognize a small set of bullet glyphs (• ○ ● ◦ plus - *), while other modules in the same crate recognize a much wider set. A list item like ▪ Item is therefore not detected as a list at all and is emitted as a plain paragraph.
Bullet glyph sets across modules (as of current main)
| Module | Glyphs | Purpose |
|---|---|---|
src/markdown/classify.rs:73-93 (is_list_item, starts_with_bullet_marker) |
• ○ ● ◦ - * |
Decides whether a line is a list item in markdown output |
src/markdown/classify.rs:130 (format_list_item) |
• ○ ● ◦ |
Converts the marker to a markdown - |
src/extractor/layout.rs:798 (is_list_marker_column) |
• ● ○ ◦ ▪ ▫ ◆ ◇ ■ □ |
Rejects spurious column-break candidates on bullet columns |
src/extractor/underline.rs:400 (is_bare_list_marker) |
• ◦ ▪ ▫ ‣ ⁃ ● ○ ■ □ - * |
Excludes bare markers from underline/strikethrough detection |
src/extractor/mod.rs:637 (is_standalone_bullet_text) |
• ○ ● ◦ |
Merges standalone bullet glyphs with following text |
src/tables/mod.rs:1319 |
• ● · |
Numeric-cell heuristics for table detection |
src/vision/fusion.rs:763 |
• ● ○ ◦ ▪ - — |
Vision fusion bullet set |
| README "Markdown output" table | • - * ○ ● ◦ |
Documented behavior |
The extractor knows that ▪ ▫ ◆ ◇ ■ □ ‣ ⁃ · are list markers (it refuses to split bullet columns on them, and excludes them from underline detection), but the markdown layer never receives that knowledge.
Reproduction
Verified on current main (added a scratch unit test, since removed):
is_list_item("▪ Item") // false ← expected: true
starts_with_bullet_marker("▪ Item") // false ← expected: true
format_list_item("▪ Item") // "▪ Item" ← expected: "- Item"
is_list_item("‣ Item") // false ← expected: true
is_list_item("◆ Item") // false ← expected: true
So a PDF list rendered with ▪ markers produces:
▪ First item
▪ Second item
▪ Third item
instead of:
- First item
- Second item
- Third item
Downstream consequence: since is_list_item() gates heading classification (src/markdown/heading.rs:291) and code classification, a short bold ▪-bullet line may additionally be misclassified as a heading.
Proposed direction
Add a single shared bullet-marker constant (e.g. pub(crate) const BULLET_MARKERS: &[char] in src/markdown/classify.rs or a common module) and use it in is_list_item(), starts_with_bullet_marker(), format_list_item(), is_list_marker_column(), is_bare_list_marker(), and is_standalone_bullet_text(), so glyph sets can no longer drift apart between modules. The extractor sets suggest the full working set is at least:
• ● ○ ◦ ▪ ▫ ◆ ◇ ■ □ ‣ ⁃ · - *
· (U+00B7 MIDDLE DOT) is intentionally ambiguous (it also appears in normal prose), so it may warrant separate treatment rather than inclusion in the automatic list-marker set.
Environment
- pdf-inspector
main(checked out today) - Only library-level unit tests; no specific PDF needed to reproduce (works from pure string inputs)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/markdown/classify.rs, especially is_list_item(), starts_with_bullet_marker(), and format_list_item(), then compare the marker checks in src/extractor/layout.rs, src/extractor/underline.rs, and src/extractor/mod.rs. Add coverage for ▪, ‣, and ◆ using the library-level unit-test setup described in the issue, and verify those markers become Markdown list items without changing ambiguous · handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- markdown, rust
- Domain
- backend, content
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100