Improve Arabic/RTL PDF text extraction: logical reading order and table column associations are not preserved

Improve Arabic/RTL PDF text extraction: logical reading order and table column associations are not preserved

Description

MarkItDown's PDF converter does not reliably preserve the logical reading order of Arabic and other right-to-left (RTL) text.

In some PDFs, Arabic characters, words, lines, and table columns are extracted according to their visual placement in the PDF rather than their Unicode logical reading order. This can produce reversed text and, more importantly, incorrect relationships between table headers and values.

I observed this behavior with MarkItDown 0.1.3 and reproduced the reversed Arabic output with 0.1.7.

Example

Expected Arabic text:

تاريخ المزاد
مزاد ربى نجد
قطعة رقم 604 شمالاً
شركة أركان الكيان العقارية

Extracted output:

دازملا خيرات
داﺰـﻣ
ﺪﺠﻧ ﻰﺑر
604 مقر ةعطق لاً امش
ةيراقعلا نايكلا ناكرأ ةكرش

This appears to involve more than simple line ordering. Some text also contains Arabic Presentation Forms, and mixed Arabic/numeric sequences are reconstructed incorrectly.

Table extraction issue

The problem becomes more significant when extracting structured information from RTL tables.

For example, the source PDF may contain property boundaries equivalent to:

Direction | Boundary -- | -- North | Plot 604 South | Plot 602 East | Plot 597 West | 30-meter street

After conversion, the RTL table order is not preserved consistently. During downstream LLM extraction, boundary values were consequently assigned to the wrong directions.

For legal or property documents, this can materially change the meaning of the source data.

Impact on downstream LLM extraction

I compared two approaches:

1. Sending the original PDF directly to an LLM.

2. Converting the PDF to Markdown with MarkItDown and sending the resulting Markdown.

MarkItDown substantially reduced token usage and request time:

• Native PDF input: 40,148 tokens

• MarkItDown input: 22,604 tokens

• Reduction: 17,544 tokens / 43.7%

• Native PDF request time: 115.44 seconds

• MarkItDown request time: 90.82 seconds

However, the extracted Markdown introduced material accuracy regressions, including:

• Arabic city name تمير being extracted or interpreted as تيمر

• A region becoming impossible to resolve correctly

• Property boundary directions being reassigned

• A plan number being malformed

• The inferred auction type changing

• Arabic descriptions containing reversed words and unnatural word order

The token savings are significant, but for documents containing legal identifiers, deed information, or property boundaries, these extraction errors can outweigh the efficiency benefit.

Reproduction

import io
from markitdown import MarkItDown, StreamInfo
with open("arabic-brochure.pdf", "rb") as file:

pdf_bytes = file.read()
converter = MarkItDown(enable_plugins=False)
result = converter.convert_stream(

io.BytesIO(pdf_bytes),

stream_info=StreamInfo(

mimetype="application/pdf",

extension=".pdf",

filename="arabic-brochure.pdf",

),

)
print(result.text_content)

The same general behavior occurs with the CLI:

markitdown arabic-brochure.pdf -o brochure.md

Expected behavior

For RTL PDFs, the converter should ideally:

1. Reconstruct Arabic text in Unicode logical reading order.

2. Normalize Arabic Presentation Forms where appropriate.

3. Preserve RTL word order without reversing the internal order of numbers, dates, URLs, or Latin text.

4. Preserve associations between table headers, columns, and cell values.

5. Correctly handle mixed-direction content such as Arabic combined with English, numbers, URLs, phone numbers, and legal identifiers.

6. Expose layout or confidence metadata when the logical reading order cannot be reconstructed reliably.

A simple character-level or string-level reversal would not be sufficient because an Arabic line may contain content such as:

رقم الرسم 604 / 2026 - https://example.com

The Arabic segments require RTL-aware reconstruction, while 604, 2026, and the URL must retain their internal LTR order.

Possible improvements

Potential approaches might include:

• Applying Unicode Bidirectional Algorithm-aware processing during text reconstruction.

• Detecting and normalizing Arabic Presentation Forms returned by the underlying PDF extraction layer.

• Using glyph coordinates and writing direction to reconstruct RTL lines in logical order.

• Detecting the dominant direction of table regions and ordering columns accordingly.

• Adding a conversion option such as:

text_direction="auto" | "rtl" | "ltr"

• Adding an RTL-specific post-processing stage for PDF extraction.

• Emitting warnings or confidence metadata when reading order is ambiguous.

• Adding Arabic/RTL PDF regression fixtures covering:

  • paragraphs

  • multi-column layouts

  • tables

  • Arabic + Latin text

  • numbers and dates

  • URLs

  • phone numbers

  • legal identifiers

Environment

• Python: 3.10

• MarkItDown: reproduced with 0.1.3; reversed Arabic output also observed with 0.1.7

• Platform: Linux

• Input: multi-page Arabic real-estate auction brochure

• Conversion: built-in PDF converter through convert_stream()

Additional information

I can provide a redacted PDF and additional source-vs-extracted examples if that would help with reproducing the issue.

The test document contains Arabic paragraphs, property-information tables, legal identifiers, dates, phone numbers, and mixed RTL/numeric content.

Start with the built-in PDF converter through convert_stream(), using the provided Python reproduction or the markitdown arabic-brochure.pdf CLI example. Compare extracted output with the expected Arabic text and table relationships, including mixed Arabic, numbers, and URLs. Done means RTL content has logical reading order, correct table associations, and preserved internal LTR sequences.