docling-project / docling-project/docling

export_to_markdown() embeds NUL bytes (\x00) for superscript unit/exponent glyphs, causing silent truncation downstream

Open
#3,858 3 comments 0 reactions 0 assignees View on GitHub
bug docling-document
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
3d 4h
Merged PRs (30d)
95

Description

## Bug description

`DocumentConverter().convert(...).document.export_to_markdown()` embeds literal **NUL bytes (`\x00`)** in the exported markdown wherever the source PDF renders a superscript numeral as part of a unit or exponent (e.g. "mL⁻¹", "m⁻³"). Conversion itself reports `ConversionStatus.SUCCESS` with no errors/warnings, so the corruption is silent.

This is a serious downstream hazard: SQLite/libSQL (and any other C-string-based storage) truncates a `TEXT` value at the first embedded NUL byte on `INSERT`. In our pipeline this silently truncated a 77,007-character document down to 1,487 characters — just the abstract — with the write reporting success and no error at any layer.

## Reproduction

```python
from docling.document_converter import DocumentConverter

result = DocumentConverter().convert("")
md = result.document.export_to_markdown()

assert "\x00" not in md # fails
```

Source PDF: a 22-page two-column IntechOpen book chapter (open access, CC-BY): https://doi.org/10.5772/intechopen.77380

## What we observed

- `result.status` → `ConversionStatus.SUCCESS`, `result.errors` → `[]`
- `len(md)` → 77007, but `md.count("\x00")` → **12**
- Every occurrence sits where a superscript unit/exponent should appear. Context around each (`\x00` shown literally):

```
'nent (30.45 -70.60 mg.mL \x00 1 ). A high n'
'thoxyphenol (5 -11 mg.mL \x00 1 ) followed '
'methylphenol (2 -4 mg.mL \x00 1 ). Wood vin'
' | SG (g.mL \x00 1 ) '
' |\n| (mg.mL \x00 1 ) 1 L. leuc'
' | \x00 2-methoxyphen'
' rubber wood (1.012 g mL \x00 1 ) and bambo'
') and bamboo (1.010 g mL \x00 1 ) were with'
'nent (30.45 -70.60 mg mL \x00 1 ). A high n'
'thoxyphenol (5 -11 mg mL \x00 1 ) followed '
'methylphenol (2 -4 mg mL \x00 1 ), Table 2 '
' the tar oil 460 at kg/m \x00 3 , was effec'
```

In every case the NUL stands in for a superscript digit that should read as `⁻¹` or `⁻³` (i.e., "mL⁻¹", "m⁻³"). This looks like a font-glyph mapping issue during PDF text extraction — the superscript minus/exponent glyph resolves to codepoint U+0000 instead of the intended character.

## Expected behavior

`export_to_markdown()` (and presumably the underlying `TextItem`/`DoclingDocument` text content) should never contain NUL bytes. At minimum, unmappable/control glyphs should degrade to something safe (e.g. dropped, or replaced with a placeholder) rather than passing through as `\x00`, since NUL is not valid character data in most downstream text pipelines and databases.

## Environment

- docling: 2.114.0
- docling-core: 2.87.1
- Python: 3.13.14
- OS: macOS-26.5.2-arm64 (Apple Silicon)
- Conversion used default `DocumentConverter()` settings (no custom pipeline options)

## Workaround

Stripping `\x00` from `export_to_markdown()`'s output before further processing avoids the truncation, but the root cause (glyph mapping to NUL during PDF text extraction) should ideally be fixed upstream so no NUL bytes are emitted in the first place.

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the supplied IntechOpen PDF using DocumentConverter().convert() and export_to_markdown(), then inspect the underlying TextItem/DoclingDocument text where superscript glyphs become NUL bytes. Trace whether the corruption occurs during PDF text extraction or markdown export; done means the reproduced output contains no literal NUL bytes while preserving the superscript content or a safe fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.