Unstructured-IO / Unstructured-IO/unstructured
bug/Wrongly detected fileType for exported documents
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 15.5k
- Forks
- 1.3k
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 13
Description
Describe the bug
I have a document exported from confluence which is downloaded as a .doc file, on trying to partition this file, getting errors as it is not able to detect the file extension. This occurs when the file is sent as byte stream and not when the file is sent as byte stream (as similar how unstructured python client SDK does this)
File partition fails with message "unstructured.partition.common.UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type." when using unstructured directly, "expected str, bytes or os.PathLike object, not int" when using client SDK
To Reproduce
from io import BytesIO
from unstructured.partition.auto import partition
with open("Test.doc", "rb") as f:
# Not directly sending the stream but sending it as wrapped bytes like how client SDK sends stream as uploaded file
elements = partition(file=BytesIO(f.read()))
Expected behavior
The extension should be detected as .doc and should return partitions.
Screenshots
NA
Environment Info
Python version: 3.10.15
unstructured version: 0.17.5
unstructured-inference version: 0.8.10
Additional context
The .doc format exported by confluence does not contain magic bytes, thus OLE file detection files and all other detection step fails, When reaching line , it fails as the file name returned is random and is sometimes an integer.
Since this is the last effort to identify the extension, can we use metadata file name before doing this check ?
This PR proposes a potential fix for the same - https://github.com/Unstructured-IO/unstructured/pull/3786
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at unstructured/file_utils/filetype.py around line 355 and reproduce the BytesIO case from the issue with Test.doc. Compare the metadata filename path with the client-SDK byte-stream path and inspect PR 3786 for prior context. Done means the .doc extension is retained and partition(file=BytesIO(...)) returns partitions without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100