Unstructured-IO / Unstructured-IO/unstructured

bug/Wrongly detected fileType for exported documents

Open
#3,980 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
HTML
Stars
15.5k
Forks
1.3k
Avg merge
4d 2h
Merged PRs (30d)
13

Description

Describe the bug
I have a document exported from confluence which is downloaded as a .doc file, on trying to partition this file, getting errors as it is not able to detect the file extension. This occurs when the file is sent as byte stream and not when the file is sent as byte stream (as similar how unstructured python client SDK does this)

File partition fails with message "unstructured.partition.common.UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type." when using unstructured directly, "expected str, bytes or os.PathLike object, not int" when using client SDK

To Reproduce
from io import BytesIO
from unstructured.partition.auto import partition

with open("Test.doc", "rb") as f:
# Not directly sending the stream but sending it as wrapped bytes like how client SDK sends stream as uploaded file
elements = partition(file=BytesIO(f.read()))

Expected behavior
The extension should be detected as .doc and should return partitions.

Screenshots
NA

Environment Info
Python version: 3.10.15
unstructured version: 0.17.5
unstructured-inference version: 0.8.10

Additional context
The .doc format exported by confluence does not contain magic bytes, thus OLE file detection files and all other detection step fails, When reaching line , it fails as the file name returned is random and is sometimes an integer.

Since this is the last effort to identify the extension, can we use metadata file name before doing this check ?
This PR proposes a potential fix for the same - https://github.com/Unstructured-IO/unstructured/pull/3786

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at unstructured/file_utils/filetype.py around line 355 and reproduce the BytesIO case from the issue with Test.doc. Compare the metadata filename path with the client-SDK byte-stream path and inspect PR 3786 for prior context. Done means the .doc extension is retained and partition(file=BytesIO(...)) returns partitions without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.