docling-project / docling-project/docling
[Bug] Docling incorrectly splits Vietnamese diacritics with spaces - causing severe text corruption__
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Problem Summary
Docling DocumentConverter produces corrupted output when processing Vietnamese PDF documents. Vietnamese words with diacritics (tone marks) are incorrectly split with spaces, making the text unreadable and unusable for RAG/LLM applications.
### Steps to Reproduce
1. Use any Vietnamese PDF document with diacritics
2. Convert using DocumentConverter:
```python
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert("vietnamese_document.pdf")
markdown = result.document.export_to_markdown()
```
3. Observe corrupted output
### Expected Behavior
Vietnamese text should be preserved correctly:
```
ĐIỀU KHOẢN & ĐIỀU KIỆN SỬ DỤNG DỊCH VỤ NGHỈ DƯỠNG
Áp dụng từ ngày 15/05/2025
Chủ thẻ Visa Infinite
```
### Actual Behavior
Vietnamese words are split with spaces:
```
ĐIỀ U KHO ẢN & ĐIỀ U KI Ệ N S Ử D Ụ NG D Ị CH V Ụ NGH Ỉ DƯỠ NG
Áp dụ ng t ừ ngày 15/05/2025
Ch ủ th ẻ Visa Infinite
```
### Impact
- **Severity**: High - Output is completely unusable
- **Affected users**: All Vietnamese language users
- **Use cases affected**: RAG systems, document indexing, text extraction, LLM training data
### Additional Context
This issue appears to be a systematic problem with how Docling handles Vietnamese Unicode characters. The corruption pattern is consistent across multiple documents tested.
Contributor guide
Assessment
This issue has not been assessed yet.