docling-project / docling-project/docling
Arabic في (fī) replaced by # in markdown serialization
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
## Description
When converting Arabic PDFs containing the high-frequency grammatical particle **في** (fī, meaning "in"), Docling's markdown serialization replaces it with `#`. This produces thousands of spurious hash characters in the output, corrupting the Arabic text.
## Reproduction
**Input:** Any Arabic-dominant PDF containing في — e.g., UAE Federal Decree-Law No. 33 of 2021 (مرسوم بقانون اتحادي رقم 33 لسنة 2021).
**Steps:**
1. Convert with Docling (`DocumentConverter` → `export_to_markdown()`)
2. Compare against `pdftotext` output from the same PDF
**Observed:**
- Docling output: **2,923 occurrences** of spurious `#` where في should appear
- `pdftotext` output from the same PDF: **162 clean في**, zero `#`
**Expected:** في should appear in the markdown output, matching the source PDF text layer.
## Example
Docling output (corrupted):
```
المادة (1) يعمل بأحكام القانون المرفق # شأن تنظيم علاقات العمل
```
Expected output:
```
المادة (1) يعمل بأحكام القانون المرفق في شأن تنظيم علاقات العمل
```
## Analysis
The substitution appears to happen during the markdown serialization stage, not during text extraction — the underlying `DoclingDocument` likely has the correct Unicode codepoints, but they are lost or substituted when converting to markdown format. The `#` character has no semantic relationship to في (U+0641 U+064A); this may be a CMap or glyph-mapping issue in the PDF backend that surfaces only at serialization time.
## Impact
- Every occurrence of في in Arabic legal/regulatory documents is corrupted
- في is one of the most common words in Arabic (comparable to "in" in English)
- Affects downstream NLP, search, and RAG pipelines that consume the markdown output
## Workaround
We currently apply a post-processing regex that detects Arabic-dominant text (>30% Arabic script characters) and replaces inline `#` (non-heading-initial, surrounded by non-whitespace) with في. This is fragile and language-specific.
## Environment
- Docling version: latest (via pip/uv)
- Python: 3.12
- OS: macOS (Apple Silicon) and Linux
- PDF type: text-layer Arabic PDFs (not scanned/OCR)
Contributor guide
Research direction
Start with the DocumentConverter → export_to_markdown() path and compare its output with pdftotext for an Arabic PDF containing في. Trace the same text through the DoclingDocument and markdown serialization; done means في is preserved in the markdown output without spurious # characters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100