Unstructured-IO / Unstructured-IO/unstructured
bug/fallback-encoding-detection-for-file-like-objects
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 15.5k
- Forks
- 1.3k
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 13
Description
Describe the bug
Currently, when trying to determine a file type from its initial text (text_head), the parser attempts fallback character-set detection only for physical file paths. For file-like objects (e.g., SpooledTemporaryFile, BytesIO streams), it blindly decodes the first 4096 bytes using the default encoding (usually UTF-8) with errors="ignore". This can strip characters and lead to corrupted text or incorrect filetype classification when ingesting documents from cloud storage (S3/GCS) or APIs.
To Reproduce
Pass a non-UTF-8 encoded BytesIO object to detect_filetype.
Expected behavior
File-like objects should use the same detect_file_encoding fallback mechanism as physical file paths to avoid data loss on decoding errors.
Screenshots
N/A
Environment Info
N/A - General bug in filetype.py
Additional context
Resolves the TODO in filetype.py:text_head.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in filetype.py at text_head and compare how physical file paths invoke detect_file_encoding with how file-like objects are decoded. Reproduce the issue with a non-UTF-8 BytesIO passed to detect_filetype, then verify that file-like objects use the fallback mechanism without losing characters or misclassifying the file type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100