docling-project / docling-project/docling
export_to_markdown() embeds NUL bytes (\x00) for superscript unit/exponent glyphs, causing silent truncation downstream
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 95
Description
## Bug description
`DocumentConverter().convert(...).document.export_to_markdown()` embeds literal **NUL bytes (`\x00`)** in the exported markdown wherever the source PDF renders a superscript numeral as part of a unit or exponent (e.g. "mL⁻¹", "m⁻³"). Conversion itself reports `ConversionStatus.SUCCESS` with no errors/warnings, so the corruption is silent.
This is a serious downstream hazard: SQLite/libSQL (and any other C-string-based storage) truncates a `TEXT` value at the first embedded NUL byte on `INSERT`. In our pipeline this silently truncated a 77,007-character document down to 1,487 characters — just the abstract — with the write reporting success and no error at any layer.
## Reproduction
```python
from docling.document_converter import DocumentConverter
result = DocumentConverter().convert("")
md = result.document.export_to_markdown()
assert "\x00" not in md # fails
```
Source PDF: a 22-page two-column IntechOpen book chapter (open access, CC-BY): https://doi.org/10.5772/intechopen.77380
## What we observed
- `result.status` → `ConversionStatus.SUCCESS`, `result.errors` → `[]`
- `len(md)` → 77007, but `md.count("\x00")` → **12**
- Every occurrence sits where a superscript unit/exponent should appear. Context around each (`\x00` shown literally):
```
'nent (30.45 -70.60 mg.mL \x00 1 ). A high n'
'thoxyphenol (5 -11 mg.mL \x00 1 ) followed '
'methylphenol (2 -4 mg.mL \x00 1 ). Wood vin'
' | SG (g.mL \x00 1 ) '
' |\n| (mg.mL \x00 1 ) 1 L. leuc'
' | \x00 2-methoxyphen'
' rubber wood (1.012 g mL \x00 1 ) and bambo'
') and bamboo (1.010 g mL \x00 1 ) were with'
'nent (30.45 -70.60 mg mL \x00 1 ). A high n'
'thoxyphenol (5 -11 mg mL \x00 1 ) followed '
'methylphenol (2 -4 mg mL \x00 1 ), Table 2 '
' the tar oil 460 at kg/m \x00 3 , was effec'
```
In every case the NUL stands in for a superscript digit that should read as `⁻¹` or `⁻³` (i.e., "mL⁻¹", "m⁻³"). This looks like a font-glyph mapping issue during PDF text extraction — the superscript minus/exponent glyph resolves to codepoint U+0000 instead of the intended character.
## Expected behavior
`export_to_markdown()` (and presumably the underlying `TextItem`/`DoclingDocument` text content) should never contain NUL bytes. At minimum, unmappable/control glyphs should degrade to something safe (e.g. dropped, or replaced with a placeholder) rather than passing through as `\x00`, since NUL is not valid character data in most downstream text pipelines and databases.
## Environment
- docling: 2.114.0
- docling-core: 2.87.1
- Python: 3.13.14
- OS: macOS-26.5.2-arm64 (Apple Silicon)
- Conversion used default `DocumentConverter()` settings (no custom pipeline options)
## Workaround
Stripping `\x00` from `export_to_markdown()`'s output before further processing avoids the truncation, but the root cause (glyph mapping to NUL during PDF text extraction) should ideally be fixed upstream so no NUL bytes are emitted in the first place.
Contributor guide
Research direction
Reproduce the issue with the supplied IntechOpen PDF using DocumentConverter().convert() and export_to_markdown(), then inspect the underlying TextItem/DoclingDocument text where superscript glyphs become NUL bytes. Trace whether the corruption occurs during PDF text extraction or markdown export; done means the reproduced output contains no literal NUL bytes while preserving the superscript content or a safe fallback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100