microsoft / microsoft/markitdown

Error processing very long words

Open
#1,488 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
186k
Forks
13.7k
Avg merge
1d 4h
Merged PRs (30d)
49

Description

We've tried to process a document from MS Marco dataset that has the word with length about 190k characters.

The processing has failed. The document is attached.

✗ Error processing C:\msmarco-v2 Top 0 - 5k files\'Longest' word has 189,819 letters, takes three hours to pronounce.docx: Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\venv\Scripts\markitdown.exe\__main__.py", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File "C:\venv\Lib\site-packages\markitdown\__main__.py", line 91, in main
    result = markitdown.convert(args.filename)
  File "C:\venv\Lib\site-packages\markitdown\_markitdown.py", line 1563, in convert
    return self.convert_local(source, **kwargs)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\markitdown\_markitdown.py", line 1587, in convert_local
    return self._convert(path, extensions, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\markitdown\_markitdown.py", line 1731, in _convert
    raise FileConversionException(
        f"Could not convert '{local_path}' to Markdown. File type was recognized as {extensions}. While converting the file, the following error was encountered:\n\n{error_trace}"
    )
markitdown._markitdown.FileConversionException: Could not convert 'C:\msmarco-v2 Top 0 - 5k files\'Longest' word has 189,819 letters, takes three hours to pronounce.docx' to Markdown. File type was recognized as ['.docx', '.wb3', '.doc', '.spo', '.opt', '.rvt', '.vsd', '.msi', '.pub', '.mtw', '.ac_', '.dot', '.pps', '.ppt', '.xla', '.xls', '.wiz', '.sou', '.wps', '.apr', '.msc', '.adp', '.db', '.wdb', '.xlr']. While converting the file, the following error was encountered:
 
Traceback (most recent call last):
  File "C:\venv\Lib\site-packages\markitdown\_markitdown.py", line 1715, in _convert
    res = converter.convert(local_path, **_kwargs)
  File "C:\venv\Lib\site-packages\markitdown\_markitdown.py", line 779, in convert
    sheets = pd.read_excel(local_path, sheet_name=None, engine="xlrd")
  File "C:\venv\Lib\site-packages\pandas\io\excel\_base.py", line 495, in read_excel
    io = ExcelFile(
        io,
    ...<2 lines>...
        engine_kwargs=engine_kwargs,
    )
  File "C:\venv\Lib\site-packages\pandas\io\excel\_base.py", line 1567, in __init__
    self._reader = self._engines[engine](
                   ~~~~~~~~~~~~~~~~~~~~~^
        self._io,
        ^^^^^^^^^
        storage_options=storage_options,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        engine_kwargs=engine_kwargs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\venv\Lib\site-packages\pandas\io\excel\_xlrd.py", line 46, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        filepath_or_buffer,
        ^^^^^^^^^^^^^^^^^^^
        storage_options=storage_options,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        engine_kwargs=engine_kwargs,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\venv\Lib\site-packages\pandas\io\excel\_base.py", line 573, in __init__
    self.book = self.load_workbook(self.handles.handle, engine_kwargs)
                ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pandas\io\excel\_xlrd.py", line 63, in load_workbook
    return open_workbook(file_contents=data, **engine_kwargs)
  File "C:\venv\Lib\site-packages\xlrd\__init__.py", line 172, in open_workbook
    bk = open_workbook_xls(
        filename=filename,
    ...<8 lines>...
        ignore_workbook_corruption=ignore_workbook_corruption,
    )
  File "C:\venv\Lib\site-packages\xlrd\book.py", line 68, in open_workbook_xls
    bk.biff2_8_load(
    ~~~~~~~~~~~~~~~^
        filename=filename, file_contents=file_contents,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        ignore_workbook_corruption=ignore_workbook_corruption
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\venv\Lib\site-packages\xlrd\book.py", line 645, in biff2_8_load
    raise XLRDError("Can't find workbook in OLE2 compound document")
xlrd.biffh.XLRDError: Can't find workbook in OLE2 compound document

'Longest' word has 189,819 letters, takes three hours to pronounce.docx

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in markitdown/_markitdown.py at _convert and the converter path reached from convert_local; the traceback shows the attached DOCX being sent to pandas.read_excel with engine="xlrd". Reproduce with the attached document and trace converter selection. Done means the document processes without the incorrect Excel-conversion failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.