docling-project / docling-project/docling

Arabic في (fī) replaced by # in markdown serialization

Open
#3,802 2 comments 0 reactions 0 assignees View on GitHub
triage/close-fixed
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

## Description

When converting Arabic PDFs containing the high-frequency grammatical particle **في** (fī, meaning "in"), Docling's markdown serialization replaces it with `#`. This produces thousands of spurious hash characters in the output, corrupting the Arabic text.

## Reproduction

**Input:** Any Arabic-dominant PDF containing في — e.g., UAE Federal Decree-Law No. 33 of 2021 (مرسوم بقانون اتحادي رقم 33 لسنة 2021).

**Steps:**
1. Convert with Docling (`DocumentConverter` → `export_to_markdown()`)
2. Compare against `pdftotext` output from the same PDF

**Observed:**
- Docling output: **2,923 occurrences** of spurious `#` where في should appear
- `pdftotext` output from the same PDF: **162 clean في**, zero `#`

**Expected:** في should appear in the markdown output, matching the source PDF text layer.

## Example

Docling output (corrupted):
```
المادة (1) يعمل بأحكام القانون المرفق # شأن تنظيم علاقات العمل
```

Expected output:
```
المادة (1) يعمل بأحكام القانون المرفق في شأن تنظيم علاقات العمل
```

## Analysis

The substitution appears to happen during the markdown serialization stage, not during text extraction — the underlying `DoclingDocument` likely has the correct Unicode codepoints, but they are lost or substituted when converting to markdown format. The `#` character has no semantic relationship to في (U+0641 U+064A); this may be a CMap or glyph-mapping issue in the PDF backend that surfaces only at serialization time.

## Impact

- Every occurrence of في in Arabic legal/regulatory documents is corrupted
- في is one of the most common words in Arabic (comparable to "in" in English)
- Affects downstream NLP, search, and RAG pipelines that consume the markdown output

## Workaround

We currently apply a post-processing regex that detects Arabic-dominant text (>30% Arabic script characters) and replaces inline `#` (non-heading-initial, surrounded by non-whitespace) with في. This is fragile and language-specific.

## Environment

- Docling version: latest (via pip/uv)
- Python: 3.12
- OS: macOS (Apple Silicon) and Linux
- PDF type: text-layer Arabic PDFs (not scanned/OCR)

Contributor guide

Open the contributing guide

Research direction

Start with the DocumentConverter → export_to_markdown() path and compare its output with pdftotext for an Arabic PDF containing في. Trace the same text through the DoclingDocument and markdown serialization; done means في is preserved in the markdown output without spurious # characters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.