docling-project / docling-project/docling

[Bug] Docling incorrectly splits Vietnamese diacritics with spaces - causing severe text corruption__

Open
#3,170 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Problem Summary

Docling DocumentConverter produces corrupted output when processing Vietnamese PDF documents. Vietnamese words with diacritics (tone marks) are incorrectly split with spaces, making the text unreadable and unusable for RAG/LLM applications.

### Steps to Reproduce

1. Use any Vietnamese PDF document with diacritics
2. Convert using DocumentConverter:
```python
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert("vietnamese_document.pdf")
markdown = result.document.export_to_markdown()
```
3. Observe corrupted output

### Expected Behavior

Vietnamese text should be preserved correctly:
```
ĐIỀU KHOẢN & ĐIỀU KIỆN SỬ DỤNG DỊCH VỤ NGHỈ DƯỠNG
Áp dụng từ ngày 15/05/2025
Chủ thẻ Visa Infinite
```

### Actual Behavior

Vietnamese words are split with spaces:
```
ĐIỀ U KHO ẢN & ĐIỀ U KI Ệ N S Ử D Ụ NG D Ị CH V Ụ NGH Ỉ DƯỠ NG
Áp dụ ng t ừ ngày 15/05/2025
Ch ủ th ẻ Visa Infinite
```

### Impact

- **Severity**: High - Output is completely unusable
- **Affected users**: All Vietnamese language users
- **Use cases affected**: RAG systems, document indexing, text extraction, LLM training data

### Additional Context

This issue appears to be a systematic problem with how Docling handles Vietnamese Unicode characters. The corruption pattern is consistent across multiple documents tested.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.