firecrawl / firecrawl/pdf-inspector
Code listings lose all newlines and indentation: a 9-line Python block extracts as one line
- Dominant language
- Rust
- Stars
- 19.1k
- Forks
- 1.3k
- Avg merge
- 9h 21m
- Merged PRs (30d)
- 51
Description
**What happens**
Converting a text-based PDF that contains a code listing collapses the whole listing
onto one line and emits it as plain text. Every newline and all leading whitespace are
lost, and no fenced code block is produced.
For Python this is unrecoverable rather than cosmetic: indentation *is* the syntax, so
the output cannot be parsed, re-indented or diffed back to the original.
Filing here rather than on `anydoc`, per the redirect in its issue config ("PDF handling
lives in pdf-inspector"). The `.docx` side of the same class of problem was fixed in
firecrawl/anydoc#35 (Source Code paragraph styles flattening into plain paragraphs).
**Reproduction**
Minimal 818-byte PDF, 9 lines of Python set in Courier, one line per `Tj`/`T*`.
Generate it with no dependencies:
```python
# mkpdf.py -> writes repro.pdf
import io
BS = chr(92)
lines = [
"# repro.py",
"def outer(items):",
" total = 0",
" for item in items:",
" if item > 0:",
" total += item",
" else:",
" total -= item",
" return total",
]
def esc(s):
return s.replace(BS, BS + BS).replace("(", BS + "(").replace(")", BS + ")")
content = ("BT\n/F1 11 Tf\n14 TL\n50 750 Td\n"
+ "".join("(%s) Tj T*\n" % esc(l) for l in lines) + "ET\n")
objs = [
"<< /Type /Catalog /Pages 2 0 R >>",
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
"<< /Length %d >>\nstream\n%sendstream" % (len(content), content),
"<< /Type /Font /Subtype /Type1 /BaseFont /Courier >>",
]
out = io.BytesIO(); out.write(b"%PDF-1.4\n"); offs = []
for i, o in enumerate(objs, 1):
offs.append(out.tell())
out.write(("%d 0 obj\n%s\nendobj\n" % (i, o)).encode("latin-1"))
xref = out.tell()
out.write(("xref\n0 %d\n0000000000 65535 f \n" % (len(objs) + 1)).encode())
for o in offs:
out.write(("%010d 00000 n \n" % o).encode())
out.write(("trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n"
% (len(objs) + 1, xref)).encode())
open("repro.pdf", "wb").write(out.getvalue())
```
Then:
```bash
python mkpdf.py
npx --yes @firecrawl/anydoc repro.pdf
```
**Actual output** — one line, no fence, no indentation:
```
# repro.py def outer(items): total = 0 for item in items: if item > 0: total += item else: total -= item return total
```
**Expected output** — line structure and indentation preserved, ideally fenced:
```python
# repro.py
def outer(items):
total = 0
for item in items:
if item > 0:
total += item
else:
total -= item
return total
```
**The data is present in the PDF**
`pypdf` recovers the listing from the same file exactly, newlines and all four-space
indents intact:
```python
from pypdf import PdfReader
print(PdfReader("repro.pdf").pages[0].extract_text())
```
So this is not a limitation of the source document — the text-positioning operators
carry the line structure and it is being discarded during conversion.
**Also seen on real documents**
Reproduced across five unrelated 38–62 page technical PDFs containing Python listings:
zero code fences emitted in any of them, every listing flattened.
**Environment**
- Reproduced through `@firecrawl/anydoc` 0.2.4 (via `npx`), which routes PDF handling here
- Node.js v22.20.0
- Windows 11
- Comparison extractor: `pypdf` 6.14.2
**Note on scope**
Fenced-block detection and language tagging are a nice-to-have. The blocking part is
newline and leading-whitespace preservation — without those the code content is lost
regardless of how it is marked up.
Contributor guide
No contributing guide indexed for this repository
Research direction
Run the provided mkpdf.py and npx --yes @firecrawl/anydoc repro.pdf reproduction first, then trace the PDF text-extraction entry point used by @firecrawl/anydoc. Compare its result with pypdf's PdfReader(...).pages[0].extract_text(); done means the nine-line Python listing retains its newlines and leading indentation instead of being flattened.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, python, rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100