microsoft / microsoft/markitdown

Incorrect parsing of Unicode smart quotes from `.docx` files

Open
#1,219 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
186k
Forks
13.7k
Avg merge
1d 4h
Merged PRs (30d)
49

Description

Bug: Incorrect parsing of Unicode smart quotes from .docx files

When using MarkItDown to convert .docx files created by Microsoft Word (default settings, smart quotes enabled), Unicode characters such as:

  • Apostrophes ( U+2019)
  • Left double quotes ( U+201C)
  • Right double quotes ( U+201D)

are incorrectly parsed and appear in the Markdown output as corrupted characters like Æ, ô, ö.

Steps to Reproduce:

  1. Create a new .docx in Word with smart quotes enabled (default setting).
  2. Add text such as: It’s important to “quote” text properly.
  3. Run MarkItDown to convert the .docx to .md.
  4. Observe corrupted characters in the output.

Expected Behavior:
Smart punctuation should either:

  • Be preserved correctly as Unicode characters, or
  • Be flattened gracefully to ASCII equivalents (' and ").

Actual Behavior:
Corrupted non-ASCII characters appear in Markdown.

Workarounds:

  • Disabling smart quotes in Word avoids the issue.
  • Alternative tools like Pandoc handle .docx smart punctuation correctly.

Environment:

  • MarkItDown version: 0.1.1
  • Python version: 3.12
  • OS: Windows 11

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with a Microsoft Word .docx containing the listed smart punctuation, then trace the MarkItDown .docx conversion entry point and its text decoding path. Done means the Markdown output preserves the Unicode characters or converts them cleanly to ASCII equivalents without corrupted characters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.